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I. INTRODUCTION 


The rapid advance of information systems 
technology and the broader spectrum of threats 
we face has caused our Command, Control, 
Communications, Computers, Intelligence, 
Surveillance and Reconnaissance (C4ISR) 
procedures and architectures to undergo 
profound changes. Joint Vision 2010 highlights 
"Information Superiority" as the major enabler 
for achieving Full Spectrum Dominance and 
C4ISR is the major contributor to achieving 
information superiority. This notwithstanding, 
the resource constrained environment of the 
Department of Defense (DoD) is forcing 
decision makers to face tough choices in 
determining the allocation of investment 
between C4ISR and the other contributors to 
force effectiveness. Cogent analyses are needed 
to support these decisions. 


Heretofore, the analysis community has not 
effectively been able to assess the relative 
contribution of C4ISR to force effectiveness vis- 
a-vis other factors. Past analyses have not been 
adequate for the new emphasis on C4ISR. The 
analytic community must now posture itself to 
address the analysis of C4ISR for 2010 and 
beyond. 


The Military Operations Research Society 
(MORS) organized and conducted a workshop 
to examine the question of Analyzing C4ISR for 
2010. That workshop was held 27-29 October 
1998 at the Center for Strategic Leadership, US 
Army War College, Carlisle Barracks, PA. The 
General Chair of the Workshop was RADM 
Robert Nutwell, USN. Captain Jay Kistler, 
USN, was the Deputy Chair and Dr. Russell 
Richards, MITRE Corporation, was the 
Technical Chair. 


1.1 Goals and Objectives of the Workshop. The 
goals of the workshop were to: 


1. Share information on the current state of 
C4ISR analysis. 

2. Determine the appropriate metrics and 
methodologies for analysis of C4ISR for 
2010. 


3. Define key areas of improvement for the 
application of analysis in support of decision 
making on C4ISR investments and 
warfighting utility. 


The objectives were to: 


ὁ Enumerate the relevant issues pertaining to 
the analysis of C4ISR in 2010. 

e Identify metrics that are sensitive to the 
effects of C4ISR on force-level 
effectiveness. 

e Assess methodologies to analyze and 
quantify the effectiveness of C4ISR 

e Evaluate appropriate tools to measure the 
benefit of C4ISR. 

e Discuss the requirements for and 
employment of advanced tools, methods, 
and research. 


1.2 Approach. The subject of C4ISR analysis 
was examined in the context of a number of 
different mission areas. A working group was 
assigned to each mission area. The mission 
areas were: 


1. Major Theater War. 

2. Smaller Scale Contingencies. 
3. Operations Other Than War. 
4. Infrastructure Assurance. 

5. Peacetime Engagement. 


In addition, separate working groups were 
assigned to the focus areas of Information 
Architectures and Analytical Techniques and 
Tools. 


1.3 Taskings. Consistent with the goals and 
objectives of the workshop, each working group 
was charged to undertake five taskings: 


Characterize C4ISR within the working 
group mission area or focus area. 

Define the relative worth of C4ISR. 
Develop and recommend Measures of Merit 
(MoM). 

Identify and describe tools. 

Analyze, synthesize and infer. 
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2. BACKGROUND 


The increasing emphasis on joint operations 
driven by the Nichols/Goldwater legislation, the 
information age and the changing military threat 
have led to significant changes in the 
Department of Defense. The Joint Warfighting 
Capability Assessments (JWCAs) are performed 
for each of 11 joint warfighting capability areas. 
Reflecting the emphasis on conducting joint 
operations, all of the warfighting capability areas 
cut across service boundaries. With most of the 
analysis tools having been developed by a single 
service to support the specific needs of that 
service, the tools available to support joint 


analysis and to support tradeoff decisions which | 


cross service boundaries are deficient. This is 
particularly true for evaluating the contributions 
of C4ISR. 


Recognizing these deficiencies, the Vice 
Chairman of the Joint Chiefs of Staff and 
Chairman of the Joint Requirements Oversight 
Council (JROC) in 1995, Admiral William 
Owens, asked the Military Operations Research 
Society to host a workshop to look at the ability 
of the community to support the JWCAs. That 
workshop was conducted October 1995. It had 
working groups for each of the eleven joint 
warfighting capability areas. Each working 
group evaluated the "health" of the analysis 
capabilities as they pertain to the capability area. 
Most reported serious deficiencies in the models, 
simulations, databases, measures and 
methodologies. This was particularly true for 
the C2TW and ISR working groups. Influenced 
significantly by that discouraging evaluation of 
the tools, Admiral Owens and other senior 
decision makers in the Department of Defense 
took steps to remedy the situation. 


Two manifestations of the actions taken to 
remedy the problem were the initiation of the 
Joint Analytic Model Improvement Program 
(JAMIP) and the C4ISR Decision Support Task 
Force (DSTF). The JWARS simulation 
development effort and the Joint Data System 
(JDS) program for improving the databases for 
supporting joint analysis were two programs that 


evolved from JAMIP. The DSTF concluded that 
the existing tools were severely deficient for 
meeting the C4ISR analysis requirements. It 
also concluded that DoD needed an organization 
focusing specifically on C4ISR analysis. That 
task force led to the creation of the C4ISR 
Decision Support Center (DSC) and the C4ISR 
Joint Battle Center (JBC). 


In the few years following the DSTF, there has 
been continuing interest and emphasis on C4ISR 
analysis. There has been much work towards 
improving the capabilities of the tools (e.g., 
JWARS, C4ISR Federation, NETWARS), and 
there have been several major C4ISR analysis 
efforts including the C4ISR Mission Assessment 
(CMA) conducted to support the Quadrennial 
Defense Review (QDR). Nevertheless, there are 
still many problems with assessing the 
contribution of C4ISR to warfighter 
effectiveness and to trade-off the contributions 
of C4ISR with other systems. This MORS 
workshop provided an opportunity to reassess 
the situation and to offer recommendations to 
our DoD Sponsors about improving the health of 
C4ISR analysis. 


3. THE WORKSHOP ACTIVITIES 


3.1 Plenary Sessions. The workshop kicked off 
the first day with four excellent presentations, 

each of which validated the need for our meeting 
and which offered challenges to the participants. 


“The Decision Maker’s Perspective.” Mr. Art 
Money, the Assistant Secretary of Defense for 
Command, Control, Communications and 
Intelligence (ASD(C3D) and the Department of 
Defense’s Chief Information Officer, started 
things off with a keynote address providing the 
decision maker’s perspective. He emphasized 
the importance of information superiority and 
acknowledged that the topics and goals of our 
workshop were timely and on track. His talk 
provided an interesting historical perspective on 
conflicts involving the United States. Mr. 
Money considered the six US security epochs 
listed in Table 1. 


Table 1. Epochs in US Security History 


— 


920-1945 
1945-1993 


1993-1998 


The first epoch covered the establishment of the 
US as a nation that was pursuing an isolationist 
foreign policy. This isolationism continued 
through the second epoch, from 1820-1900, 
when the US was involved with the Civil War 
and later becoming a world power. The third 
epoch, from 1900-1920, was characterized by 
World War I, the collapse of Europe and the 
birth of communism. The fourth epoch, from 
1920-1945, witnessed massive technological 
advancement, and included World War II, the 
defeat of Fascism and the development of the 
atomic bomb. Epoch number five, from 1945- 
1990, includes the Korean War, the Vietnamese 
War, the Cold War, the eerie predictability of a 
monolithic threat and bi-polar powers. 


Gap 


The sixth security epoch, from 1990 to the 
current time, presents a new paradigm. The 
collapse of the Soviet Union as a world power, a 
major desert war and the emergence of many 
turbulent forces have highlighted it. These 
include the 500-700 year old grudges between 
often well armed nation states, technological 
advancement and commercial expansion, the 
proliferation of WMDs, and threats to the US 
that include quasi-governmental organizations 
(Bin Laden), transnational entities and tribes. 
The threats are more asymmetric than prior 
threats and include threats to the US 
infrastructure and cyber attacks. 


This perspective demonstrated the shift in the 
defense requirements that has taken place during 


SY 2  ὃὦὲ νςἝἬ 
[1820-1900 -|CivlWar Ἐὃοὃὄοὃό δ... 
WWI: Birth of Communism 

WWII: Fascism defeated, atomic bomb, massive technology advances 
(aircraft, communications, radars, UHF, etc.) 

Cold War: monolithic threat (Soviet Union), Vietnam, Korea, ICBM, Fulda 
, SAC; predictabilit 
Coalition War: collapse of a major power, walls come down, transnational 
threat, non-government powers, no geographic boundaries, asymmetric 
warfare, critical infrastructure protection, information warfare, sensitivities 
of information, operations other than war 


our country’s history and demonstrated why 
there is increasing interest in C4ISR. 


Mr. Money emphasized the importance of 
information superiority — providing the night 
information to the right people in the right 
format at the right time. However, he also 
emphasized the need to assess the Return On 
Investment (ROD for C4ISR systems — 
precisely the reason for the MORS workshop. 
He challenged the workshop to determine what a 
pound of C4ISR is worth. He stated that we do 
not presently do this well, and we need better 
tools and methodologies to evaluate the 
contribution that C4ISR makes to force 
effectiveness. He specifically highlighted the 
need for better models and simulations and for 
better measures. 


3.1.2 “The Operator’s Perspective.” Mr. 
Money’s talk was followed by two presentations 
providing the operator’s perspective. The first 
was given by Brig Gen Carol Elliott, the Vice 
Director for the Joint Staff J-2 and the Team 
Lead for the Intelligence, Surveillance and 
Reconnaissance (ISR) Joint Warfighting 
Capability Assessment (JWCA). She was a 
voice of experience, for she has been grappling 
with the problem of determining the ROI for 
ISR during the last couple of years, most 
recently the Recce 2010 Study. She echoed the 
remarks of Mr. Money in indicating that the 
analysis community faces big challenges in 
determining the ROI for ISR, and in 
acknowledging that we do not do this very well. 


She stated that the JWCA process has a 
voracious appetite for people who can bring 
coherent analysis to the table. She stated that 
analysis of the value of C4ISR would require 
innovative, new approaches. 


General Elliott gave a pithy discussion of 
analysis pitfalls derived from her own 
experiences in supporting the ISR JWCA. Some 
of the points that she made were: 


e Make sure that terms such as "Dominant 
Battlespace Awareness" are well defined. 

¢ Make sure that all analysis assumptions are 
made clear to the decision makers. 

e The use of notional data is a real problem. 

e Be sure the analysis does not exclude people 
as part of the ISR process. 

e Be sure that the measures of merit are 
relevant (it is not just a problem of 
mathematics). 

9 Itis always difficult to determine how much 
analysis is enough. 

9 Whatever the results, you will be misquoted 
misinterpreted, attacked, disregarded or all 
of the above. 

e Keep the "I" in ISR. It is more than a 
question of platforms. 

e Models are not adequate for the types of 
tradeoff analyses that they are being asked 
to do. Models used for ISR analyses were 
developed for other things. Better measures 
are needed to assess the contribution of ISR. 


> 


General Elliott closed with five 
recommendations for C4ISR analysis: 


Define effectiveness. 

Replace assertions with facts. 

Scrutinize results for what’s missing. 
Explain your results to your administrative 
staff to see if your explanation passes the 
common sense test. 

5. Remember that it’s people, stupid! 


oo πο 


Colonel William Cooper, representing the Joint 
Staff J-6, provided the second operator’s 
perspective. Like General Elliott, Colonel 
Cooper is a JWCA team leader — for the 
Command and Control area. Colonel Cooper’s 


presentation focused on the problems with the 
legacy stovepiped tools for addressing C2 issues. 
He discussed the need for service models to 
interoperate so that joint issues can be 
addressed. Colonel Cooper also stressed the 
need for more than M&S to support C4ISR 
analyses. He discussed the need to coordinate 
joint experimentation activities of USACOM J-9 
with the modeling and simulation. He saw the 
Opportunity to validate models with 
experiments. He closed by challenging the 
community to develop better models for C4ISR 
and to tie weapons systems and C4ISR together. 


3.1.3 Technical Perspective — “C2 
Assessment: Past, Present and Future.” Dr. 
Stuart Starr, MITRE Corporation, completed the 
day’s invited presentations with a historical 
perspective on the changes that he has seen over 
the past 25 years with respect to command and 
control assessments. Dr. Starr discussed three 
“ages”: 


e Awakening 
e Dark Ages 
© Renaissance 


Prior to 1975, there was general insensitivity to 
C2 issues. C2 was often ignored or assumed to 
be perfect or, if considered, it was often treated 
simply as a patch (e.g., an additional term in 
Lanchester’s equations). The little bit of 
analysis of C2 that was done was fragmented 
within the Department of Defense. The decade 
from 1975-1985, the Awakening, saw an 
increased interest by the senior leadership in C31 
leadership as there was an increased awareness 
of the importance of (31 and a need to justify 
expenditures on C3I. This increased interest 
was manifest through several key initiatives 
which included symposia and workshops (at 
least one sponsored by MORS on (31 Measures 
of Effectiveness), the creation of new 
organizations (e.g., the C2 Research Program 
and the Naval Postgraduate School C3 
Curriculum), major multi-year studies focusing 
on C3] and increasing resources devoted to C3I 
assessment. New assessment methodologies and 
processes such as the Modular C2 Evaluation 
Structure (MCES), Headquarters Effectiveness 
Assessment Tool (HEAT), Mission Oriented 


Approach (MOA) and new tools such as 
synthetic environments like the Identify Friend 
or Foe Network (IFFN) Testbed greatly 
improved the ability to assess the effectiveness 
of C3I. Much of the credit for the Awakening 
was due to the strong leadership of people like 
Robert Hermann and Harry Van Trees. 


The Awakening period was followed by a six 
year period from 1985-1990, the Dark Ages, in 
which DoD leadership showed a significant 
reduction in interest in C3I assessments. This 
was due to relatively high budgets for (91 and 
an emphasis on C3I system acquisition. 

Because resources were plentiful, there was little 
concern about return on investment for C3I. The 
influence of analysis organizations waned 
significantly during the Dark Ages. 


The Dark Ages were followed by the 
Renaissance period (1991—Present). This was 
brought about by a combination of factors which 
included profound geopolitical changes (e.g., the 
dissolution of the USSR and the Warsaw Pact), 
severe resource reductions and changing 
paradigms such as the Revolution in Military 
Affairs (RMA) and Simulation Based 
Acquisition (SBA). Some of the manifestations 
of the renewed interest in analysis and C31 
assessments were the creation of the Defense 
Modeling and Simulation Office, the 
revitalization of the CCRP, the C4ISR Decision 
Support Task Force, the creation of the C4ISR 
Decision Support Center (DSC) and the Joint 
C4ISR Battle Center (JBC), the C4ISR Mission 
Assessment (CMA), the initiation of the Joint 
Analytic Model Improvement Program 
(JAMIP), the development of major analysis 
simulation models which made major 
improvements in the representation of C4ISR 
(e.g., JWARS, EAGLE, ARES, C4ISR Model 
and NSS) and the establishment of the Joint 
Experimentation Directorate within the US 
Atlantic Command (USACOM). MORS has 
played a major role in the Renaissance in C4ISR 


assessment. The following MORS-sponsored 
workshops have contributed to the renaissance: 


e C3IEW MOEs 

e VV&A for M&S 

e JROC Analysis 

e Advanced Distributed Simulation for 
Analysis 


e OOTW Analysis Methods and Techniques 
e Quick Response Analysis Methodologies 
(QRAM) 
e Warfare Analysis and Complexity (“New 
Sciences’) 
QDR Lessons Learned 
SIMTECH 2007 


Two NATO Panels made significant 
contributions to the renaissance during this 
period. The NATO Panel 7 Working Group 
conducted several meetings aimed at assessing 
the impact of C3I on the battlefield. The NATO 
Studies, Analysis and Simulations (SAS) Panel 
published a Code of Best Practices (COBP) for 
Assessing C2 in 1998. The COBP 
recommended an Assessment Methodology, 
which consisted of the following elements: 


Problem Structuring 

Human Factors and Organizational Issues 
Scenarios 

Measures of Merit 

Tools and Their Applications 

Data 

Risk and Uncertainty 

Reporting 


Dr. Starr then referenced General Larry Welch 
in making the case that DoD leadership must 
initiate and sustain some major cultural changes 
to track the other changes that have occurred if 
our C4ISR analyses are to be relevant. Table 2 
describes the areas in which cultural changes are 
needed and the nature of the changes. 


Table 2. Needed Cultural Changes 


Institutional 


Organizational 


Dr. Starr closed his presentation with a 
discussion of selected recommendations from 
the COBP, a characterization of the C2 
assessment base and some major challenges. 
Since many of his recommendations and 
challenges reappeared in some of the briefouts, 
we will fold in his recommendations and 
challenges with those of the working groups 
later in this report. 


3.1.4 “Federated Analysis of C4ISR in 
Warfare.”’ On the second day, Mr. Charles 
Taylor, C4ISR Decision Support Center, gave an 
interesting luncheon presentation sharing some 
of his recent experiences in grappling with 
C4ISR analysis problems at the DSC. Mr. 
Taylor made a strong point that no single model 
will be adequate to meet the needs of the C4ISR 
analysis community. He advocated the use of a 
federation of M&S tools to support the analysis. 


3.1.5 “Blackhawk Down.” Mr. Mark Bowden, 
a reporter for the Philadelphia Enquirer, gave a 
luncheon presentation on the third day. His 
multi-media presentation described the gripping 
real-life activities surrounding the attempt to 
rescue the US crew of a helicopter that crashed 
in Mogadishu. His presentation highlighted 


Bureaucratic 
izati Collaborative 
M&S Subject matter 
[Processes | Opaque ιιὀ  ὀὠηἠἪγΤιδηβρατοηί 
Data Limited to “validated” __| Include possibilities 6. 
i Allow for non-linear 
Processes | Uncertainty | Suppress | ΠΙυπηπαίο 
Force-on-force 
Scope 


mmetrical 


imited 


Narrow (force structure; 
military worth of 
equipment) 


From 


Subject matter expert 


ilitary-social-economic 
Asymmetrical 


Broad (Forces; 
equipment; doctrine; 
concepts; C2; ...) 


some of the critical problems with our C4ISR 
systems. 


3.2 Conduct of the Working Groups. After the 
first day’s plenary session, the participants broke 
into their respective working groups. In addition 
to addressing the taskings of the Workshop, each 
working group was instructed to evaluate the 
health (in the mission area) of the state of C4ISR 
analysis. 


3.3 Synthesis Team. As has become customary 
for MORS workshops, a Synthesis Team was 
created to provide a holistic view of the 
deliberations of all of the working groups 
participating in the workshop. Consistent with 
that goal, the team included representatives from 
OSD, each of the services, industry, FFRDCs 
and academia. Selected members of the 
Synthesis Team actively participated in each of 
the working groups to help the groups keep on 
track with respect to meeting the objectives of 
the workshop and to report on progress or 
difficulties encountered by the groups. The 
Synthesis Team convened periodically during 
the course of the workshop to discuss and 
compare the evolving findings and 
recommendations of each working group. 


3.3.1 Products of the Synthesis Team. The 
Synthesis Team drew on its periodic 
deliberations and its observations of the 
activities in the working groups to develop some 
broad observations about the nature of the 
problem and to formulate a set of cross-cutting 
findings and recommendations. The products of 
the Synthesis Team are summarized below. 


3.3.2 Broad Observations. As a consequence of 
monitoring the deliberations of the individual 
mission-oriented working groups it was clear 
that there was broad variability in the maturity 
of the individual mission areas (e.g., level of 
domain knowledge; available tools and data; 
experience in analyzing the C4ISR systems and 
processes associated with those missions). 
Based on the perceptions of the Synthesis Team, 
it was concluded that one would rank the 
missions on a qualitative “maturity scale” as 
follows (most mature to least mature): 


Major Theater War (MTW) 

Smaller Scale Contingency (SSC) 
Operations Other Than War (OOTW) 
Peacetime Engagement (PE) 
Infrastructure Assurance (IA) 


Bi i 


This maturity ranking should not be surprising 
as it tracks with the total amount of effort 
devoted to C4ISR analysis for the different 
mission areas. 


The Synthesis Team further observed that many 
of these mission areas subsume qualitatively 


different sub-missions. For example, Operations 
Other Than War (OOTW) subsumes 
Humanitarian Assistance (HA), disaster relief, 
peace keeping and peace enforcement. These 
sub-missions are so different in their nature and 
maturity that it can be misleading to making 
overarching statements about the overall mission 
area. For example, some of the sub-missions in 
OOTW would be rated very immature. 


The Synthesis Team also noted problems with 
definition for several of the mission areas. 
Notably, the distinction between SSC and 
OOTW was not initially clear. To eliminate 
confusion, the chairs of these two working 
groups “defined” their areas of focus to make 
the distinction. Furthermore, the Peacetime 
Engagement Working Group spent a significant 
amount of time crafting a definition of PE. 
Finally, because infrastructure assurance was a 
relatively new mission area, it was necessary for 
the co-chairs to clearly define IA. 


3.3.3 Major Issues. As the Synthesis Team 
monitored the deliberations of the individual 
working groups, it identified several major 
issues that were raised by those working groups. 
The major issues are summarized in Table 3. 
Note that, although there are considerable 
differences in those issues, several crosscutting 
issues are apparent. These crosscutting issues 
involve difficulties associated with data and 
Measures of Merit (MoMs). These specific 
crosscutting issues are discussed in the findings 
and recommendations that follow. 


Table 3: Major Issues 


Mission Area 


Peacetime Engagement 


Infrastructure Assurance 


Analytical Tools and 
Techniques 


σπσπσπσπσππ,, ΐπτττιο τ Μαοῖς..ροὃ})Εὃοὲ'ὗΉΊ 
Existing data does not map to DPG scenarios 
Data; Softness of MoMs (e.g., public opinion) 
OOTWs are not all the same 
1 Definition of Peacetime Engagement 
Data (much of it is Blue data); Higher-level MoMs; 
Scope of problem/ability to decompose meaningfully 
Dealing with the interactive nature of the problem 
Cognitive modeling; Broad array of missions; 


Size of the trade space; Representation of Red data 
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Each working group assessed the current state of 
the health of C4ISR analysis in its mission area. 
The template that was developed for the 
forthcoming NATO COBP for Assessing 
Command and Control (and which was 
introduced by Dr. Stuart Starr in his plenary 
presentation) was used for the working group 
assessments. Table 4 provides an ‘aggregate’ 
stop light assessment (Green = good; Yellow = 
fair; Orange = significantly deficient; Red = 
poor), aggregated qualitatively across the results 
of the individual working group assessments. 


Table 4. Aggregate Assessment of the Health of 
C4ISR Analysis 


Organization 


Τὸ <2 


O 
Risk and Uncertainty 


Note that none of the elements were rated Green. 
Also, two of the critical steps (Data and Tools) 
were assessed as being Red, and five of the steps 
(Problem Structuring, Human 
Factors/Organization, Measures of Merit, 
Treatment of Risk and Uncertainty and Report 
Documentation and Availability) were assessed 
as being Orange. Only the Scenario Generation 
step was rated as good as Yellow. 


These observations suggest concern about the 
ability of the analysis community to adhere to 
the processes in a strawman COBP. These 
results led the Synthesis Team to highlight 
findings and recommendations in five areas: 
COBP, three key elements of the candidate 
COBP and the overall relationship between the 
producer and consumer of C2 analyses. 


3.3.4 Findings. The findings of the Synthesis 
Group are discussed below. 


COBP. The NATO COBP provides an 
interesting and useful first step towards 


developing a sound foundation for C4ISR 
analysis. However, the scope of the COBP was 
restricted to C4ISR analysis in the context of 
conventional warfare. As this workshop 
revealed, the problem of analyzing C4ISR for 
2010 will involve a host of additional mission 
areas, which pose unique problems for the 
C4ISR assessment community. Thus, an 
expanded, community-endorsed COBP is 
needed to support the C4ISR assessment 
problem, in all its dimensions. 


Measures of Merit (MoMs). Over the last 
thirteen years, MORS has conducted several 
workshops on the issue of developing measures 
of merit for command and control assessments [ 
Ref. 1, 2]. Those workshops proposed the 
concept of a hierarchy of MoMs that would 
range from measures of system performance 
through measures of mission effectiveness. The 
various mission-oriented working groups at this 
workshop appeared to support this conceptual 
approach to the problem. However, it was 
concluded that the hierarchy must be extended to 
deal with the needs of new world disorder 
missions. The Architecture Working Group also 
observed that it lacked MoMs to support the 
effective assessment and comparison of 
architectural options. In particular, it called for 
MoMs that would relate system architectural 
characteristics to operational outcomes. 


Data. A decade ago, at a MORS Workshop on 
Simulation Technology 1997 (SIMTECH 97), 
Dr. Walt LaBerge observed that “without data 
we are nothing” [Ref. 3]. That thought clearly 
prevailed in this workshop. Each working group 
highlighted data deficiencies. The specific 
concerns varied by Working Group and included 
the following challenges: 


1. Acquiring the Blue system data needed to 
perform vulnerability assessments of critical 
infrastructures. 

2. Acquiring the data appropriate to some 
currently ill-defined future military 
operation. 

3. Characterizing Red’s C4ISR systems and 
critical infrastructures. 

4. Gaining access to needed data in a timely 
fashion. 


5. Collecting and making better use of data 
emerging from Service and Joint Advanced 
Warfighting Experiments. 

6. Validation and certification of data. 


Tools. Many of the speakers at the workshop 
focused their remarks on modeling and 
simulation and its important role in C4ISR 
analysis. It is clear, however, that no single 
class of tool (and certainly no single tool within 
a class) can satisfy C4ISR assessment needs 
adequately. There is a broad spectrum of tools 
that a C4ISR analyst can choose from, that vary 
with respect to the time to create and apply the 
tools, the cost to create and apply them, the 
amount of abstraction and their credibility (see 
Ref [4]). 


Relations between Providers and Consumers 
of C4ISR Assessments. Many Working Groups 
observed that the relationships between the 
providers and the consumers of C4ISR 
assessments are strained. By relying on 
complex, opaque M&S tools, many providers 
are failing to make their analyses transparent and 
understandable (e.g., “Why did I come to that 
conclusion? My model told me!”). Similarly, 
consumers are frequently not “educated 
customers.” For example, they fail to articulate 
issues in a way that is amenable to responsible 
C4ISR analysis. Or, they fail to give the 
provider adequate resources to either create the 
needed tools or to perform the assessment. 


3.3.5 Recommendations. This section provides 
recommendations, that the Synthesis Group felt 
would improve the state of C4ISR analysis. 
Each of the listed findings (Section 3.3.4) are 
addressed. 


COBP. Although the preliminary review of the 
NATO COBP was promising, we recommend 
that a more thorough, in-depth assessment of the 
product should be undertaken to see if it meets 
(at least some of) the needs of the US C4ISR 
analysis community. MORS is in the best 
position of any US organization to organize and 
execute such an assessment. 


If such an assessment is undertaken and it 
endorses the NATO COBP as a preliminary 


product, the following steps should be pursued. 
First, the product should be disseminated 
broadly to the C4ISR assessment community, 
along with any caveats that emerged through the 
assessment process. MORS has an efficient 
distribution network for making such 
information broadly available. Second, 
consideration should be given to employing the 
product in defense universities, such as the 
Naval Postgraduate School (NPS) and the Air 
Force Institute of Technology (AFIT), to educate 
young analysts. 


In view of the limited scope of the NATO 
COBP, efforts should be supported to extend the 
product to address new world disorder issues. 
NATO is forming Studies, Analysis and 
Simulation (SAS) Panel-015, to explore the 
feasibility of extending the initial COBP to deal 
with issues associated with OOTW and 
information operations. Those efforts should be 
supported by MORS (e.g., by providing 
representatives to the deliberations; appropriate 
briefings and case studies to the participants; a 
peer review of the panel’s product) to ensure 
that the resulting efforts are of greatest utility to 
the US C4ISR assessment community. 


Measures of Merit. To respond to the needs of 
new world disorder missions, efforts are needed 
to formulate the upper levels of the MoM 
hierarchy. For example, in the mission areas of 
smaller scale contingencies and operations other 
than war, measures are needed to characterize 
political-institutional-social stability. For 
example, are the bodies of government 
functioning effectively? Are educational 
institutions operative? Are children playing on 
the soccer fields? Similarly, in the area of 
infrastructure assurance, MoMs are needed to 
reflect public confidence in key infrastructures 
(e.g., confidence in the finance and banking 
sector as measured by the value of the Dow 
Jones Industrial Average and the value of the 
dollar versus the Euro). These needs suggest 
that MORS should convene a workshop on 
expanded C4ISR MoMs for new world disorder 
missions. 


To satisfy the needs of the architectural 
community, steps should be taken to explore 


options to develop MoMs that can be used to 
assess and compare architectural options. One 
possible course of action would call for OSD to 
convene an Integrated Process Team (IPT) of 
stakeholder organizations to address this issue. 
MORS could provide the facilitators to conduct 
such an event. 


Data. The depth and breadth of the data 
concerns are such that a new institutional 
mechanism is needed to deal with the totality of 
the problem. This mechanism should deal with 
the full life cycle of the data problem, to include 
data standardization; data acquisition; 
verifications, validation and certification; 
transformation (into a form that is useful to the 
C4ISR assessment community); and data storage 
and access. Currently, there are a number of 
organizations that are involved with significant 
facets of the problem. These include OSD’s 
C4ISR Decision Support Center (DSC), PA&E’s 
Joint Data System (JDS), Defense Modeling and 
Simulation Office (DMSO) Modeling and 
Simulation Operational Support Activities 
(MSOSA), and J6’s Joint Defense Information 
Infrastructure Control System (JDIICS) and 
NETWARS activities. OASD(C3]) should take 
the lead in forging these fragmented efforts into 
a complete, mutually reinforcing solution to the 
total problem. 


Tools and their Application. It is important that 
C4ISR analysts be aware that a broad mix of 
tools exists beyond the usual simulation tools, 
and it is important that the analysts be 
conversant about their strengths and weaknesses. 
In particular, where appropriate, C4ISR 
assessments should take better advantage of 
analyses based on basic operations research and 
physical principles; structured frameworks for 
analysis; expert elicitation tools (e.g., 
groupware); wargames; analytical systems 
dynamics models; applications of the “new 
sciences” to deal with complex adaptive 
systems; and multifunctional tools (e.g., tools 
that can support analyses and mission rehearsal). 
In all instances, assessors should assiduously 
pursue efforts to perform responsible levels of 
Verification, Validation and Accreditation 
(VV&A) for the tools selected. MORS should 
lead the way in determining what is a 
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“responsible level of VV&A.” [Note: the 
SIMVAL 99 workshop that MORS is hosting in 
1999 should shed light on this issue.] 


Because each type of tool has strengths and 
weaknesses, it 1s important to select and 
orchestrate a complementary set of tools. In the 
past, this concept has been limited to the model- 
test-model paradigm. However, it would be 
appropriate to generalize that concept to 
incorporate a broader set of tool types. For 
example, it might be desirable to consider an 
“expert elicitation-real world experience-test- 
model” paradigm, that expands the set of tools 
that are orchestrated. 


Provider-Consumer Relations. The MORS 
Mini-Symposium on “Quick Reaction Analysis 
Requirements and Methodologies (QRAM)” 
[Ref. 4] recognized and addressed the issue of 
provider-consumer relations. That workshop 
recognized that there should be a “contract” 
between providers and consumers, recognizing 
the mutual responsibilities of both sides. It is 
recommended that the draft “contract” be 
updated and refined to reflect the lessons learned 
from this workshop. As a further step, a COBP 
should be developed (perhaps in the form of a 
check list) to help educate consumers about the 
attributes of sound C4ISR analysis. This “check 
list” could be patterned after the COBP that the 
AIAA Information and C2 System Technical 
Committee is developing for OSD on C2 
experimentation. 


3.3.6 Synthesis Group Summary. The situation 
with respect to the current state of C4ISR 
analysis runs the gamut from bad news to good 
news. The bad news 15 that the C4ISR 
assessment problem 15 getting substantially more 
difficult. We are confronted with a host of new 
world disorder missions and issues for which we 
lack the key elements of effective C4ISR 
analyses (particularly relevant data and tools). 
In addition, these issues are in a current state of 
flux, making it difficult to get traction on the 
problem. One of the most dramatic examples of 
this challenge is the infrastructure assurance 
problem, where the infrastructures and the 
threats to them are changing dramatically. More 
bad news is that currently, we generally do not 


do a satisfying job in performing C4ISR 
analyses. Although there are counter-examples 
to this statement, it is recognized that most 
studies of conventional conflict fail to live up to 
the NATO COBP. 


The good news 15 that we are beginning to 
understand a great deal more about the C4ISR 
assessment problem. We are starting to 
recognize the importance of C4ISR in the 
context of the many missions that DoD must 
perform. We are also starting to recognize what 
we don’t yet understand and the practices that 
should be followed to perform credible C4ISR 
analyses. Finally, this workshop itself provides 
some more good news. If the recommendations 
of the workshop are implemented successfully, it 
will provide the foundation for the C4ISR 
community to work collaboratively to attack the 
most critical of our shortfalls. 


4. INDIVIDUAL WORKING GROUP 
SUMMARIES 


The above sections discussed the observations, 
findings and recommendations of the Synthesis 
Group aggregated over all of the working 
groups. The sections below provide similar 
products for each of the working groups. Each 
section concludes with the assessment of the 
working group as to the health of C4ISR 
analysis in the mission or specialty area. The 
discussions below represent only a summary of 
the key points from the deliberations of the 
working groups. More details are provided in 
the complete annotated working group briefings 
that are found in the appendices. 


4.1 Major Theater War (MTW) Working 
Group. The MTW working group was co- 
chaired by Mr. Charles Taylor, C4ISR Decision — 
Support Center and Dr. Mark Youngren, MITRE 
Corporation. Although the MTW area is 
certainly the most mature of the mission areas 
with respect to analysis attention, the working 
group still found serious deficiencies in its 
assessments. 


4.1.1 Analysis Setup and Problem Definition. 
The group felt that the most meaningful part of 
the analytical process is the clear articulation of 


the questions and working with the decision 
maker to ensure the question is properly 
understood by all and framed so that an answer 
can be derived. It is essential to clearly define 
the objectives and constraints and to have good 
scenarios and data. 


4.1.2 Models and Simulations. Models and 
simulations are the workhorse tools for C4ISR 
analysis in the MTW area, but the legacy M&S 
tools are, for the most part, models that were not 
built to analyze Joint problems and they do not 
model most aspects of C4ISR very well. For 
example, it is usually the case that doctrine is 
hard wired; C2 is scripted; there is limited 
explicit representation of C4ISR systems; 
limited ability to map the contributions of 
C4ISR systems to combat outcomes; fusion is 
represented poorly (if at all); human factors are 
not represented at all; Red C2 is not reactive to 
Blue actions; COMINT and HUMINT are not 
modeled well; and the flow of information 15 
poorly represented. 


Other serious deficiencies in our existing M&S 
tools are the modeling of non-combat, Red 
C4ISR, decision heuristics for Red rules of 
engagement, information operations, the play of 
coalitions and the representation of the effect of 
maneuver on Red behavior. 


Current state of the art. The working group 
raised concerns about the ability of the various 
simulations under development (e.g., JWARS) 
to significantly improve on the above 
deficiencies. They believe that much research is 
needed in areas such as perception 
development, fusion, human factors, decision 
making and on reflecting C4ISR outcomes into 
military effects like maneuver. The “best of 
breed” philosophy (selecting the best of the 
existing algorithms) of JWARS and other 
simulation development efforts offers little 
promise that those models will meet the needs of 
the C4ISR analyst. 


The working group spent a considerable amount 
of time discussing the relative advantages and 
disadvantages of four analysis approaches: 


(1) Single monolithic campaign model that is 
responsible for representing everything. 

(2) Hierarchy of models. 

(3) Federation of models. 

(4) Techniques other than simulation. 


On balance the group supported the federation 
approach as having the most promise for 
analyzing the contribution of C4ISR on 
battlefield outcomes. The federation approach 
allows fidelity where fidelity is needed, and it 
allows the analytical team to review the cause 
and effect relationships as they unfold at the 
seams of the individual models. However, the 
approach still falls prey to such problems as 
ageregation/disaggregation, inconsistent model 
treatments and (VV&A). 


The group discussed non-simulation 
methodologies such as spreadsheets, systems 
dynamics analytical models, wargames, 
exercises, historical analyses, surveys, man-in- 
the-loop simulations, but it felt that constructive 
simulations would continue to be the main tool 
for C4ISR analysis in the future. 


4.1.3 Measures of Merit. The problem is not 
the definition of C4ISR MOPs or battle MOEs. 
Rather, it is the mapping of C4ISR MOPs into 
battle MOEs, the establishment of direct 
linkages between the C4ISR system 
performance measures and the criteria for 
mission effectiveness. The mapping is very 
dependent on context. Analysis requires 
multiple measures; no single MOP or MOE 
satisfies as a global metric for performance or 
effectiveness across all mission areas, nor is it a 
sound analytical practice to measure only a 
single attribute as the criterion in a study effort. 
4.1.4 Scenarios. High level scenarios such as 
the Defense Planning Guidance (DPG) do not 
specifically address the totality of the variables 
associated with setting up an analysis. This 
requires the analyst to create portions of the 
scenario for a particular effort (e.g., specific 
force laydown, enemy behavior, own force 
ROE, etc.). All of these assumptions result in a 
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multiplicity of scenarios eventually being 
created from what is believed by senior leaders 
to be a baseline from which all analysis efforts 
depart. Another serious problem with most 
scenarios is the fact that deployment timelines 
for C4ISR assets are rarely represented. Thus, 
C4ISR asset movements into theater are rarely 
accounted for — tending to overstate 
performance. The group also questioned if it 
was appropriate in today’s environment to 
continue to focus on full blown MTW events 
rather than the more likely multiple smaller scale 
regional contingencies. 


4.1.5 Experiment Design. There is a serious 
problem with the sheer number of variables to 
be considered in an analysis and the resulting 
overflow in dimensionality. The common 
approach to deal with the problem has usually 
been to reduce (somewhat arbitrarily) the 
number of variables for which the analysis will 
consider. In so doing, the analytical team runs 
the risk of not completely exploring the cause 
and effect relationships and dependencies 
between multiple variables. The trade-off is 
obviously time and money. More research is 
needed as to how one effectively manages the 
Overflow in dimensionality. Unfortunately, most 
analysts lack an appreciation for the statistical 
issues around the experimental design and 
interpretation of analysis results. 


4.1.6 Data. The group considered three types of 
data: system performance data, scenario data 
and behavioral data. They judged the systems 
data to be the most mature; however future 
systems data are lacking. Scenario data are 
incomplete in its representation of Red forces 
especially in terms of Red C4ISR systems and 
Red behavioral data. The existing data don’t 
necessarily map into DPG scenarios. Another 
weakness is the absence of background data 
(civilian traffic, background communications 
load, etc.). The behavioral data is the weakest 
link. There are virtually no reliable data sources 
for C2 events that require decision making to 
occur or data that suggests what courses of 
action may transpire from equivalent battlefield 
events. Much research is needed for this area. 
The group felt it was necessary to have better 
standardization and cataloguing of data. 


4.1.7 Human Factors and Organization. The 
group felt that this area was the weakest 
element. Behavior models are not developed; no 
Red behavior models are available; data for 
behavior and decision making are not available; 
human-in-the-loop approaches are not repeatable 
and may not be verifiable; and we don’t know if 
we should model doctrine, ‘“‘real-world,” 
normative or optimal decisions. 


4.1.8 Risk and Uncertainty. Uncertainty is 
often modeled with probability distributions. 
However, we don’t understand the models, 
much less the distributions, much less the 
distributional parameters. 


4.1.9 General Comments. Some of the most 
critical issues surrounding studies have little to 
do with the traditional analysis activities. 
Instead, they revolve completely around 
exogenous factors that frequently end up 
determining the results out of hand. These 
variables include the time allowed to complete 
the effort, the resources provided, the politics of 
both the question to be answered and the 
organizations chosen to run it (as well as those 
left out) and the expectations of the decision 
makers surrounding the results. These issues in 
a strict scientific sense should have no bearing 
on the results and yet they frequently are the first 
order effects that determine the answer. 


4.1.10 The Bottom Line. Much basic research is 
still needed. The major deficiencies are: 


e Linkage between C4ISR effects and 
campaign outcomes. 

Fusion. 

Perception development. 

Decision making. 

Basic combat itself (e.g., maneuver effects). 


JWARS and other new model efforts are 
incorporating the “best” current algorithms — 
this approach will not work with respect to 
C4ISR. We recommend developing a prioritized 
list in the areas of C2, information fusion, 
communications, ISR, basic combat and C4ISR 
effects on combat. Use that prioritized list to 
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direct the research plan. This research needs to 
be implemented in new model developments, 
but the effort should not be focused on a 
particular tool. 


There are fundamental data gaps that hinder the 
ability to support C4ISR analyses. We 
recommend that the data requirements to support 
C4ISR analyses be identified, collected and 
made available to the analysis community. 


C4ISR analysis will require a range of different 
models (it will not be limited to JWARS). There 
is still a serious need for additional development 
of the analysis infrastructure (to include data, 
scenarios, tools and experienced analysts) to 
respond to current demands. 


4.1.11 Assessments. 


Assessment of the Health 
of C4ISR Analysis for MTW 
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4.2 Smaller Scale Contingencies (SSC) 
Working Group. In order to differentiate itself 
clearly from OOTW, the working group decided 
to focus its efforts on mission types in which the 
use of military force was likely (raids, hostage 
rescue, opposed Non-combatant Evacuation 
Operations (NEO), etc.). The missions were 
also viewed as those for which military 
considerations tend to take precedence over 
political considerations. The group recognized 
that this is an artificial distinction. Many formal 
definitions of OOTW explicitly include SSC. 
Moreover, SSC intended as nearly pure military 
operations often are changed into more political 
activities. The scenario played out in Haiti 
(which went in a matter of a few minutes from 
an invasion of a small country to a friendly 


occupation with primarily political goals) 
provided a good example of this phenomenon. 
Finally, as the group stressed in its deliberations, 
SSC tend to have significant political 
dimensions, even when the operations 
themselves are largely military in character. The 
working group also concluded that analysis of 
SSC cannot be meaningfully conducted on the 
group of missions as a whole. Their diversity 
requires that tools and analytic efforts be 
focused on meaningful, coherent subsets of the 
topic. For example, studies of raids will require 
different tools and data than studies of hostage 
rescues. Meaningful analyses must have 
baselines for assessment of the contribution of 
current and future C4ISR systems, and the 
baselines do not exist for most of these mission 


types. 


4.2.1 Characterization of C4ISR in SSC. The 
dominant requirement will be for more rapid and 
agile intelligence, planning and execution 
processes. This means exploring ways to 
improve the speed and responsiveness of the 
entire ISR process (from tasking through 
dissemination) as well as to couple situation 
awareness, decision making and execution more 
closely. SSC are different from MTW in that 
they almost always involve considerable 
pressure for centralized control. This arises both 
from the legitimate concerns of higher command 
arising from the potentially strategic 
implications of military activities down to the 
tactical level and the more idiosyncratic fact that 
SSC are often "the only game in town,” which 
means that decision makers at all levels are free 
to invest time in them. MTW, on the other hand 
are often large and complex enough that senior 
political and military officials often are too busy 
to focus on tactical level decisions and actions. 


2 


SSC may (but do not always) require a high 
level of tactical security (particularly in the 
planning, preparation and entry phases) and 
management of highly visible consequences. 
The experience of conducting an operation that 
is a military success yet fails in terms of US 
policy was all too familiar to the working group 
participants. 
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Communication issues arose several times 
during the discussion. SSC often require new or 
novel infrastructures that overcome gaps arising 
from military service differences, the need to 
include coalition partners or host nations, or the 
need to deal with non-military actors (inter- 
agency or NGO). Moreover, secure 
communications may be needed to achieve 
adequate operational security. 


SSC are not necessarily a lesser-included case of 
MTW. This is crucial when data is being 
selected, models chosen, parameters estimated 
or research is being organized. Some of the 
distinctions were mentioned earlier. Others 
include the fact that the MOFE and MOPE are 
usually not under the control of the military. In 
this sense, SSC are properly understood to be 
OOTW. Moreover, the group stressed the fact 
that the intelligence required in many SSC is 
more fine grained and detailed than that required 
for the same levels of command in MTW. 


SSC are often conducted under restraints that 
would not be present in MTW. Rules of 
engagement and political restrictions often limit 
the activities available for mission 
accomplishment. Moreover, SSC tend to be 
characterized by a very meaningful tension 
between efforts to avoid mission failure and 
maximizing the likelihood of mission success. 
Examples of decisions to limit the size and 
armament involved in operations in order to 
avoid undesirable political consequences were 
offered by several members of the working 


group. 


4.2.2 Assessing Relative Worth. The method 
for assessing the relative worth of C4ISR 
systems was seen as readily available from 
fundamental operations research techniques. 
This is not to say that the data and tools needed 
to conduct those analyses are available. The 
group recognized the value of a particular 
C4ISR innovation could be seen as a function of 
the range of SSC missions to which it applies, 
the relative likelihood of those mission types and 
their estimated impact on the relevant MOFE 
and MOPE. 


4.2.3 Measures of Merit. The group readily 
accepted both the set of measures of merit 
categories defined by MORS in the 1980s and 
the suggested change offered by Dr. Stuart Starr 
during his plenary address, of adding an outside 
layer, Measures of Policy Effectiveness 
(MOPE). This addition leads to the following 
hierarchy of measures: dimensional parameters, 
MOP, MOE, Measures of Force Effectiveness 
(MOFE) and MOPE. Equally important, the 
group also wanted to ensure that analyses of 
C4ISR in SSC "be informed by" metrics from all 
the relevant layers. Interoperability issues also 
arose several times. Metrics for interoperability 
and for information dissemination remains 
important in assessing future C4ISR systems. 


The quality of the information available was 
seen as the single most important metric for 
SSC. This was defined to include its 
responsiveness to warfighters, including the 
Commander's Critical Information Requirements 
(CCIR), but also the needs of key staff members 
and the units executing the operation. 
Responsiveness was seen as including much 
more than mere availability. 


What is meant by high quality information? 
"Perfect" information in a military situation 15 
seen as complete (for the command's needs), 
current, correct, precise enough for the 
associated military purposes (which will vary 
from targeting, which is very precise, to general 
maneuver planning, which is often much less 
precise) and consistent. Consistent here means 
both internally consistent within the data, 
information and knowledge of the C4ISR system 
and consistent across command nodes. 


Measures of merit that focus on rapid and robust 
decision making, planning and execution will be 
needed. This means we need valid and reliable 
indicators of the speed of the C4ISR processes, 
the variety of futures and alternative courses of 
action they can and do consider, and the quality 
of the decisions they make. Indicators of 
decision quality essentially must be MOFE and 
MOPE, though MOE for C4ISR, MOP for 
C4ISR systems and the dimensional parameters 
that drive them may also be needed for 
diagnostic purposes in any given analysis. 
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4.2.4 Tools. This area was perhaps the least 
mature of the areas discussed in the working 
group. The most important tool requirement 
was for credible support to Course Of Action 
(COA) analysis. This means tools that generate 
projections of alternative futures that are 
credible to operators as well as the capacity for 
sensitivity or "what if" analyses. These tools 
must also be fast enough to support mission 
planning and have an analytic mode that permits 
large numbers of runs in a short period of time. 
Moreover, these tools must be designed to pull 
from operational databases, not just notional or 
analytic ones. 


Perhaps the most novel idea arising for new 
tools was a call for a virtual C4ISR rehearsal 
capability that would be designed both as a tool 
for analysts and for operators. Operators would 
use it for training and mission rehearsal. 
Analysts could use it for improving their 
understanding of missions, generating 
hypotheses about C4ISR improvements, 
examining man in the loop elements and testing 
hypotheses. 


The group heard a similar plea for more 
effective ways to both link and differentiate real 
and simulated data, particularly in real 
headquarters participating in virtual exercises 
and training. 


In general, tools are needed to assess the degree 
to which C4ISR systems are linked across 
functions, echelons and time. New information 
systems are linking operators more and more 
closely, but no tools to measure these changes or 
their impacts are currently available. Similarly, 
models that capture the effects of "soft factors" 
such as differences in language, culture or 
training are not available but would be valuable 
in an era that stresses "jointness" and coalition 
operations. 


Analysts, particularly those supporting 
operational organizations, also felt a need to 
capture and replay information (data, voice 
traffic, imagery, etc.) for current and future 
operations. This would greatly facilitate after- 
action analyses from both operations and 
exercises. 


4.2.5 Analysis of Four SSC Mission Areas. 
The group focused in detail on four different 
types of SSC mission areas to link future C4ISR 
needs, the analytical tools required to support 
their development, the most crucial measures of 
merit, and implied key metrics. The four 
mission areas were: 


1. Raid. 

2. Urban Warfare. 

3. Counter Terrorism/Operational Disruption. 
4. Opposed/Potentially Opposed NEO. 


The discussions of measures was focused on 
MOFE and MOPE. 


4.2.6 The Bottom Line. The current and future 
C4ISR needs for SSC must be addressed within 
the context of each specific mission category, as 
opposed to addressing a broad range of similar 
missions. By heeding this, proper analysis of 
each mission area may be secured. C4ISR data 
within SSC will need to develop a higher level 
of tactical detail and control then other arenas of 
warfare (i.e. peacekeeping, peace enforcement, 
major theatre war.) This characteristic, 
developed from tight operational constraints, 
will aid commanders' tactical planning, control 
and security. Investment in such C4ISR 
information systems such as virtual rehearsal 
programs, course of action analytical tools, and 
relational databases is recommended to increase 
military force flexibility and performance. 


4.2.7 Assessments. 


Assessment of the Health of 
C4ISR Analysis for SSC 


|Problem Structuring [Orange 
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4.3 Operations Other than War. As was done 
by the SSC working group, the OOTW Working 
Group began their deliberations by scoping the 
definition of OOTW to exclude any mission in 
which combat might be anticipated. The 
working group believed the fact that OOTWs are 
going on near-continuously offered the 
opportunity to focus analysis efforts on real 
operations — observe and analyze the 
operations as they occur. This reduces the need 
to build models and simulations. 


4.3.1 Problem Decomposition. The OOTW 
group decomposed the problem. For a set of 
OOTW missions, they derived a set of typical 
functions within the missions and identified 
typical mission scenarios for analysis. For the 
scenarios, they considered how C4ISR means 
were employed in the mission. By crossing the 
C4ISR means with the mission functions, they 
identified the most interesting C4ISR evaluation 
issues. Finally, these led to a categorization of 
typical measures of merit and the appropriate 
tools to assess them. 


The decomposition effort partitioned the 
problem into 12 major functions associated with 
OOTW missions: security operations, police 
operations, refugee operations, resettlement, 
compliance, inspection, surveillance, PSYOPS 
(Perception), civil affairs, sustenance, medical 
operations and governance. They then mapped 
the functions to the following subset of OOTW 
missions: 


Peace Keeping (PK) 

Peace Enforcement (PE) 

Peace building 

Humanitarian Assistance/Disaster Relief 
(HA/DR) 

Civil support 

Nation assistance 

Counter dmg 

Counterinsurgency support 


They then focused on two different missions: 
PK/PE and HA/DR and characterized each. 

This activity demonstrated clearly that the issues 
for the different missions were significantly 
different. For example, PK/PE focuses on 


preventing conflict, restoring governance and 
stabilizing the situation, whereas, HA/DR 
focuses on stopping the suffering and dying. 
Similarly, there are different types of 
participants (coalition forces, civilian 
authorities, non-government organizations, etc.), 
levels of force, different data required and 
different measures. 


4.3.2 Tools. The working group decided that 
many different types of tools would be needed to 
address the spectrum of OOTW missions. These 
include: Expert elicitation, polling, statistical 
analysis, data mining, wargaming, collaborative 
decision making, network analysis, functional 
capability models, learning from real-world 
activities, historical analyses, analytic 
frameworks and predictive capability models. 
Of course the group felt that the usual 
simulations would also be useful, but they would 
be less important for supporting analyses of 
C4ISR in the OOTW mission area than in those 
mission areas that focused on military 
operations. 


While an end-to-end modeling approach in 
OOTW is well beyond the present state of the 
art, there is a clear need for tools and models 
that will do part of the job within the context of 
an analytic framework. For example, tools for 
HA/DRs that help to predict damage and 
casualties and generate requirements for medical 
supplies and personnel are needed. Within 
peace operations, models for assessing the 
performance of different ISR capabilities or 
communications networks would be helpful. 
Tradeoffs frequently will be based on expert 
judgment and will be qualitative rather than 
quantitative. In this area, qualitative trade-off 
judgments may be better than judgments based 
on trying to force the numbers to fit the problem. 


Trades of C4ISR with forces for OOTW require 
better definition of missions and requirements, 
more organized exploitation of past lessons 
learned, databases of operational data, 
frameworks for analysis and appropriate 
supporting tools. 


4.3.3 Measures. OOTW mission measures are 
known but may vary widely by mission. The 
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measures are standard for many trades within 
C4ISR (e.g., command and control vs 
communications), but are very difficult for 
making tradeoffs between elements of C4ISR 
and forces. Certainly, by the nature of the 
mission, the measures needed for the latter 
tradeoff were not the usual attrition measures. 


4.3.4 Recommendations. Establish an effort to 
advance our knowledge of OOTW and C4ISR in 
OOTW leading to best practices and analytic 
frameworks. Continue efforts to identify 
analytic tool requirements to assess C4ISR 
support to OOTW. Continue development of 
methods to better integrate OOTW analysis into 
DoD planning and programming. Establish a 
common syntax and semantics to facilitate a 
comparison between study results. Develop a 
deliberate approach to data collection (identify 
and prioritize needs, establish a mechanism for 
collecting data, establish repositories and 
include non-DoD sources. 


4.3.5 Assessments. 


Assessment of the Health of C4ISR 


Risk and Uncertainty 
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4.4 Peacetime Engagement Working Group. 
The Peacetime Engagement (PE) working group 
immediately came face to face with a definition 
problem. There is no current Joint Staff 
definition of PE. The group discussed the 
relationship between PE and overseas presence, 
and decided on the following definition which 
characterizes PE as deterrent in nature. 


Analysis for OOTW 
[ll a 
Organization 
[Measures «| Orange ὁ ὁ ὀἀ᾽ 
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port 


Definition. Peacetime Engagement is the 
application of resources (including forces) in 


peacetime to make shooting less likely. 
Successful PE would promote other goals: 


e Stability of commerce. 

e Deterrence. 

e Separation of potential belligerents. 
e Exercises with (potential) allies. 


Information superiority was judged to be the 
principal element of C4ISR implementation for 
the future. Successful management of 
information will depend upon development and 
establishment of joint doctrine, including 
processes and organizations. 


4.4.2 Measures of Merit. The working group 
devoted much of its time discussing measures of 
merit. They partitioned the measures as falling 
into five categories: 


Level of effort. 
Task performance. 
Mission progress. 
Political goals. 
Costs. 


APY 


They discussed a variety of measures within 
each category, some of which were standard 
measures like number of reconnaissance flights 
flown and ISR reports delivered. However, 
many were measures unlike those used for 
combat operations. For example, the price of oil 
and the number of civilian truck deliveries. To a 
large extent the measure of success for a PE 
mission is the extent to which daily activities are 
routine. 


4.4.3 Insights. The group discussions generated 
several useful insights about C4ISR analysis in 
the PE mission area. Some of the insights were: 


e Information operations highest utility will be 
in PE. 

e Intelligence Preparation of the Battlefield 
(IPB) is conducted during PE. 

e PE is nota lesser included case of warfare. 

e Coalitions are important; there will be 
greater numbers of potential partners in PE 
than in specific MTWs. 
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e Interoperability of C3] systems will be 
critical for PE because of the many potential 
partners. 

e PE requires drawing in political 
constraints/considerations to a degree 
greater than for other mission areas. 


4.4.4 Data. Like the other mission areas, the 
working group judged the availability of data to 
be a serious problem. The significant role of 
political constraints generates peculiar data 
requirements. Furthermore, the uncertainty 
about the probabilities of specific threats creates 
significant data collection and distribution 
needs. 


4.4.5 Assessments. Because PE is not very well 
defined, the state of the practice for analysis 
related to any aspect of PE has many gaps. 


Assessment of the Health of C4ISR 
Analysis for PE 


Human Factors/ Yellow 
=r) veggies 
|Measures——s| Orange Cds 
|Data——“‘wL Orange sd 


Riskand Uncertainty |Red ὁ ὃ9΄ΌΘΘῚ 
|Report | Orange 


4.5 Infrastructure Assurance. Since 
Infrastructure Assurance (IA) is a relatively new 
and evolving mission area addressing a new kind 
of warfare unlike what we have experienced in 
conventional warfare, the working group spent 
its initial discussions defining the mission area 
and being educated about recent Presidential and 
DoD activities related to the area. Following is 
the official definition of Critical Infrastructure 
Protection (CIP) (CIP and IA were considered to 
be synonymous) provided in the Presidential 
Decision Directive (PDD) 63, signed in May 
1998. 


4.5.1 Definition. Infrastructure assurance 15 the 
planning to improve the readiness, reliability and 
continuity of infrastructures such that they are: 


e Less vulnerable to disruptions or attack. 

e Harmed to a lesser degree in event of 
disruption or attack. 

e Can be readily reconstituted to re-establish 
vital capabilities. 


The DoD mission areas for IA are divided into 
11 infrastructure sectors (3 of which comprise 
the C4ISR mission area): defense information 
infrastructure, command, control and 
communications, intelligence, surveillance and 
reconnaissance, public works, financial services, 
transportation, health affairs, emergency 
services, personnel, space and logistics. 


The sectors cannot be considered in isolation; 
each is dependent on the others to achieve 
mission objectives. Since these services are 
routinely provided by commercial assets outside 
the government sector, the focus of analysis 
must broaden to include law enforcement, Other 
Government Agencies (OGA) and the private 
sector. The dependencies among the sectors 
must be well understood for analysis. 


4.5.2 DoD Approach to Infrastructure 
Assurance. The draft DoD CIP Plan states that 
the approach to JA is to: 


e Identify relevant characteristics of each 
infrastructure that is critical to military 
mission success. 

e Analyze military plans to identify critical 
infrastructure assets. 

Assess vulnerabilities of critical DoD assets. 
Redirect resources to reduce prioritized 
vulnerabilities. 

e Collaborate with other government agencies 
and the private sector to reduce 
vulnerabilities when services are provided 
from outside the DoD fence. | 


The above steps are conducted in the context of 
six lifecycle phases that occur before, during, 
and after events which may compromise or 
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disrupt the services or mission accomplishment 
of a critical infrastructure: 


Analysis and assessment (preventive). 
Remediation (preventive). 

Indications and warning (before an event). 
Mitigation (before and during an event). 
Response (during an event). 
Reconstitution (after an event). 


4.5.3 Data. Operational architecture data on 
organizational missions, functions, structures, 
systems and their relationships to include 
information flows and dependencies are needed. 
This presents unique problems because much of 
the needed data must come from the private 
sector and are proprietary. Much inter-agency 
information sharing is needed to satisfy 
requirements. The MDITS database for counter- 
terrorism is one example of a database that must 
be shared with the IA community. 


4.5.4 Analysis Tools. Current DoD analysis 
tools generally lack the capability to facilitate 
detailed IA analysis because C4ISR 
requirements generally have not addressed the 
need to support IA. IA tools have been 
developed in response to [A community 
requirements and are effective for analyzing 
commercial infrastructure and the military 
missions that they support. Integrated and 
interoperable tools or comprehensive models for 
analyzing defense infrastructures are non- 
existent. Likewise tools to conduct 
interdependency analyses are non-existent. 
Some of the tools that are used for analysis 
supporting IA are the Joint C4ISR Architecture 
Planing/Analysis System (JCAPS) and the NCS 
for communications. For cyber infrastructure, 
the NIPC and various Certs are C4ISR 
components that support analysis and 
assessments. Wargaming is frequently used to 
better understand the problems and issues. 


4.5.5 Measures of Merit. The working group 
suggested a hierarchy of measures of merit for 
IA and its dependence on C4ISR as a basis for 
mission success and national security. The 
hierarchy (from top to bottom) is: 


National security. 
Missions (DoD and other). 
IA functions. 


C4ISR systems performance. 


4.5.6 Issues. Some of the major issues 
identified by the working group are: 


e Getting governmental entities within DoD 
and other Federal, state and local agencies to 
share information (which is sometimes 
proprietary), to share costs, and to have a 
common understanding of IA. 

Fidelity of data. 

Getting the commercial sector to provide 
proprietary data. 

Developing a central data repository. 
Security and privacy issues. 

Legal issues. 

International cooperation. 


4.5.7 Recommendations. The group made two 
recommendations: 


1. MORS continue to emphasize IA as an 
essential element of C4ISR (and other) 
analyses and wargaming and simulation 
exercises. 

2. MORS and the IA community begin to 
develop mutual understanding of the 
requirements of IA analysis requirement in 
order to develop integrated or 
complementary tools. 
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4.5.8 Assessments. 


Assessment of the Health of C4ISR Analysis 
for Infrastructure Assurance 
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4.6 Information Architectures. The 
Architectures working group was included as 
part of the workshop because of the increasing 
emphasis and importance of operational and 
systems architectures for information systems to 
the C4ISR analyst. Analytical paradigms have 
traditionally focused on evaluating the 
effectiveness of a platform or system. 
Architectures provide a structured approach for 
describing and understanding a system-of- 
systems associated with a specified domain, 
such as C4ISR. In so doing, this allows that 
domain to be considered as if it were an entity. 
Using the architecture construct enables our 
analytical focus to evolve from being platform- 
centric to being architecture-centric. 


Architectures are still an evolving discipline 
which demands a combination of both art and 
science. They help us understand the 
relationships and dependencies among tasks, 
operational elements accomplishing those tasks 
and the information flow that needs to occur 
among the operational elements to accomplish 
the tasks appropriately. They enable us to 
describe the systems and communications that 
support the information flow across nodes and, 
in so doing, enable the understanding of how 
information technology supports work. The 
architecture descriptions form the basic data set 
of activities, operational elements, information 
flows, nodes, systems and communications that 
support a wide variety of C4ISR issue analyses. 


An architecture should not be viewed as an end 
in itself, but should be a means to an end. 
Operational architectures can capture 
operational requirements in a fashion that allows 
consistent traceability to design specifications. 


4.6.1 Keys to Realizing the Potential of 
Architectures. The group discussed several 
things needed to realize the potential of 
architectures. The keys are summarized below: 


e Miust be able to access, manipulate and reuse 
data across a wide range of stakeholders to 
include developers, users and acquirers. 

e Must be able to integrate human factors into 
the information architectures. People are an 
integral part of any information architecture. 

e Need to represent the threat more overtly to 
include analysis of adversary as well as 
friendly. 

e Better tools are needed (better visualization 
tools, more flexible tools, more usable by 
the operational subject matter experts). 

e Must be able to better articulate the 
operational impacts of architectural analysis 
to warfighters and to decision makers. 


4.6.2 Measures of Merit. Measures are needed 
for information architectures to quantify the 
relationship between measures of performance 
and operational measures of effectiveness. 
Measures are also needed to evaluate an 
architecture. Three aspects that could be 
measured are: 


1. How well the architecture description has 
been developed. 

2. To what extent is the architecture 
description being used. 

3. The value of the architecture in achieving 
C4ISR goals and objectives. 


4.6.3 Assessments. Because information 
architectures do not represent a mission area, the 
standard COBP for evaluating the health of 
C4ISR Analysis is not as relevant for 
architectures as it 1s for a mission area. 
Nevertheless, the architectures working group 
provided an assessment (with perhaps some 
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differences in the interpretations of the 
elements). 


Assessment of the Health of C4ISR Analysis for 
Information Architectures 


Organization | 
FScenarios | Yellow, ὁ 
[Measures | Red 
Yellow/Orange 
b 


Risk and Uncertaint 


4.7 Analytical Techniques and Tools. Even 
though each of the other working groups 
addressed the issue of tools to support C4ISR 
analysis within a specific mission area, MORS 
had a separate working group focusing 
specifically on analytical techniques and tools 
for C4ISR analysis. The group discussed a 
variety of methodologies to include simulations 
such as JWARS, network analysis models such 
as NETWARS, optimization tools, systems 
dynamics models and Complex Adaptive 
Systems (CAS). However, the focus was on the 
workhorse simulation tools. 


4.7.1 Taxonomy of Decision Issues. In order to 
structure the assessments and to facilitate 
communication, the working group developed a 
taxonomy of decision issues. The taxonomy 
partitioned the C4ISR analysis problem along 
three dimensions according to the category of 
decision issue (force structure, cognitive 
behavior, CONOPS, architecture, 
vulnerabilities, trade space), the decision maker 
(acquisition, operational, POM/budget) and 
scale of conflict (MTW, SSC, OOTW). Then, 
within each element of the cube the working 
group developed a qualitative assessment of the 
ability of existing tools and techniques to 
support C4ISR analysis. This qualitative 
assessment considered tools and techniques in 
ageregate (simulations, optimization, CAS, 
systems dynamics, game theory, etc.) and 
considered a variety of evaluation criteria 
(scenario, data, human behavior, maturity, 


measures, etc.). For many of the cells 
(combinations of decision issue by decision 
maker by mission area, comments were provided 
to explain the reasons for the assessment. The 
number of combinations are too voluminous to 
be presented in this summary; however we will 
provide some general observations based on the 
evaluations. See the briefing slides from the 
Analytical Techniques and Tools Working 
Group in the Appendix for the all of the 
evaluations provided. 


All of the assessments for the operational and 
acquisition decision makers were either yellow 
or red (no green). The situation was judged to 


be somewhat better for the POM decision maker © 


with a sprinkling of green assessments for the 
CONOPS decision category and for force 
structure (the area where we have the most 
experience) under the MTW mission area. The 
most frequent explanation as to why an 
evaluation was rated yellow or red was a 
problem with data. 


4.7.2 The Sky is Not Falling. Even though the 
evaluations of the existing analytical techniques 
and tools painted a bleak picture as to providing 
support to the C4ISR analysis community, the 
group emphasized that the story is not as bad as 
it may sound. There are some promising new 
analytical techniques; there are some promising 
new simulation development efforts; and there is 
a good core of analytical talent. 


4.7.3 Recommendations. The working group 
was optimistic that the problems could be fixed 
with additional attention. Specific 
recommendations were to: 


9 Continue to fund research and development 
in the applications of current methods and 
new tools. 


Refine joint analysis tools. 

Train analysts to do analysis of joint issues. 
9 Emphasize more effective ways to explore 

the scenario space. 


5. CLOSING COMMENTS 


This workshop purposely sought out individuals 
who were not among the “usual MORS 
workshop attendees” for the express purpose of 
seeking out fresh new ideas about C4ISR 
analysis. To a large extent the memberships 
(and the chairs) of the Information Assurance 
and the Information Architectures Working 
Groups were comprised of people who were not 
regular MORS workshop attendees. There was 
widespread agreement that this expanded 
membership contributed greatly to the success of 
the workshop. Thanks should go especially to 
the chairs of those working groups: Mr. Thomas 
Bozek and Mr. Frank Ruggeri for Information 
Assurance and Ms. Patsy McGrady for 
Information Architectures. MORS should 
continue the practice of seeking out "new blood" 
for workshops. 


Spectrum of Analysis Techniques. Analysts 
will always have to rely on clever, innovative 
use of the entire spectrum of evaluation 
techniques. Table 5 depicts the most common 
techniques and characterizes each according to 
resources, lead time, breadth of application, 
replicability and credibility. It shows seven 
classes of techniques ranging from expert 
elicitation to real crises and combat. A mix of 
tools will generally be required to compensate 
for the shortfalls of individual tools and to 
adequately represent Blue and Red processes. 


Table 5. Spectrum of Evaluation Techniques. 


Real Crises/ 
Combat 


A mix of tools will also be necessary to pursue 
an iterative approach to C4ISR analysis. The set 
of techniques appropnate for a broad and 
shallow initial cut at the problem will be 
different from the techniques appropriate for 
subsequent narrower and deeper cuts at the 
problem. In the early stages, the techniques will 
tend to be those at the lower part of the spectrum 
(expert elicitation, wargames, analytical 
models). Quick and inexpensive techniques will 
be needed to efficiently explore the scenario 
space, to determine what are the critical 
parameters and to design the analyses that will 
follow in the later stages. In the later stages, the 
techniques will tend to be more resource 
intensive constructive, virtual or live 
simulations. 


5.2 Comments on the Self-Assessments. The 
COBP template proved to be very useful for 
focusing discussions during the workshop. The 
assessments of the working groups about the 
health of the state-of-practice for C4ISR analysis 
were not encouraging, but I suspect that we 
tended to be overly pessimistic in our 
assessments. Yes, many improvements are 
needed — particularly with respect to human 
factors, scenarios, data and analysis tools. 
However, there was little mention during the 
workshop of perhaps the most important element 
in the conduct of good C4ISR analysis — the 
analyst. 


5.3 The Importance of Good Analysts. We 
need to keep in perspective that even good tools 
and data and scenarios do not guarantee good 
analyses. Instead, good analyses result from: 


e The ability to parse a problem down to its 
essence. 

e The ability to synthesize the diverse parts of 
a problem. 
Domain expertise. 
Availability of analysis tools (including data 
and scenarios). 

e Knowledge of the tools and the craft of 
analysis. 
Experience on the job. 
Knowledge of military operations. 
Good communications between the analyst 
and the consumer. 


Most of the above requirements for good 
analyses point to the need for good, experienced 
analysts. Analysts knowledgeable about their 
tools can frequently figure out ways to make 
good use of the available tools. Good analysts 
can make innovative uses of tools to address 
questions that the tools were never designed to 
address. If good tools are placed in the hands of 
good analysts, the combination is unbeatable. 
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Gary Federici, CNA, stated in his response to 
my inquiries about tool to support C4ISR 
analysis a few years ago: 


I have an uneasy feeling that many of 
those who are involved in all this are 
not clearly aware of some crucial 
truths about modeling. There is a 
widespread perception, I believe, that 
a model is a black box which accepts 
assumptions on the input side and 
reports outcomes as output. ... What 
relatively few people reflect on, from 
lack of direct experience in or study of 
campaign analysis, is that within the 
black box sit a group of analysts who 
pull its levers and turn its dials and 
have a very major influence on its 
outcomes. 


It is absolutely essential that DoD pay as much 
(or more) attention to developing and retaining 
good analysts. In the words of Dr. Stuart Starr, 
DoD needs to develop an “intellectual reservoir” 
upon which to base future C4ISR analysis 
activities. Such a reservoir will require a 
continuing, highly capable, multidisciplinary 
team with access to all of the relevant 
information. The C4ISR Decision Support 
Center is one model of a government-run 
organization. Another potential organizational 
model is the Phase One Engineering Team 
(POET), which draws on personnel from 
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FFRDCs and not-for-profits to support analyses 
of ballistic missile defense issues. 


5.4 Some Encouraging Signs. In spite of the 
many deficiencies in the C4ISR analysis area, 
there are several encouraging signs. First, the 
analysis community realizes that there are many 
deficiencies and that steps need to be taken to 
remedy the deficiencies. High-level decision 
makers are aware of the deficiencies, and they 
seem willing to invest the resources to address 
the problems. 


Secondly, efforts are already underway to 
remedy some of the deficiencies. For example, 
JWARS, NSS, NETWARS and the C4ISR 
Federation are all major programs to develop 
improved decision support tools. The M&S 
community is beginning to understand that a 
new paradigm in modeling is necessary with 
explicit consideration given to modeling the 
information and decision processes, joint 
operations, communications, fusion and 
perception management. However, this 
workshop has pointed out that no single 
framework, no single tool, no single measure 
was sufficient to deal with the full set of issues 
of interest. The challenge is to select and 
orchestrate the needed tools and measures to 
reflect the substantive issues in question and to 
be responsive to the needs of the decision 
makers. 
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Major Theater War: 
Working Group Agenda 


Tuesday Afternoon: 
— 1300-1430 Introductions and Opening Remarks (3-5 Min. 
per person) 
— 1500-1530 Chair stakes out mission and ROE 
— 1530-1630 Problem Articulation For Next Day 
Wednesday Morning 
— 0800-0900 Analysis of the Current State of the Art 
— 0900-1000 New Methodologies and Tools 
— 1000-1100 Requirements for C4ISR Analysis 
Wednesday Afternoon 
— 1230-1330 Synthesis of Morning 
— 1330-1600 Recommendations 
— 1600-1630 Wrap-up 
Thursday Morning 
— 0800-1000 Recap/ New Topics 
— 1000-1200 Write Up Findings As Group 
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Major Theater War: 
Approach 


° What is the Scope of this Working Group? 
— Definitions 


© What is the current State of the Art? 
— Models 
- Analysis 


° What are the appropriate tools to “give DoD the capability to do C4ISR 
Analysis?” 


Approach: 


The approach detailed on this and the next slide were developed beforehand by the 
working group co-chairs. The group agreed to follow this general approach. 
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Major Theater War: 
Appreach (cont’d) 


© What are the requirements for C4ISR Analysis? 
— Models 
-- Data 
— Analysis architectures 
— Supporting infrastructure 


e What are the recommendations for the Workshop Sponsors? 


¢ The Bottom Line 
— Qutbriefing 
— Final report 


Major Theater War: 
Scope of the Working Group 


Consider issues related to producing analysis in support of major DoD 
programmatic decisions (e.g., QDR) 
~ Analysis of C4ISR programs and issues (e.g., CONOPS, 
interoperability) 


— Analysis of other programs and force structure in JV 2010, to include 
shortcomings 


Examine how we can do the analysis, which includes, but is much more than, 
developing models of C4ISR 


Scope of the Working Group: 


The first substantive task of the group was to reach a consensus on what matters were 
properly within the purview of the group, and what matters were not. The group agreed 
that the focus would be on the use of C4ISR analysis and tools to support major 
programmatic decisions such as the QDR. This would include the analysis of C4ISR 
systems in the context of total force structure, but would also include the representation 
of C4ISR in the analysis of force structure and military sufficiency, again in the context 
of QDR-like analysis. The proper use of C4ISR analysis and models in support of CINC 
COA determination (in particular CENTCOM and USFK MTW modeling) was 
considered to be outside of the scope of the group. 


Some particular comments and issues raised by the group in this discussion were: 


e Is an MTW scenario enough to make statements? The DPG IPS are useful to size the 
force structure but not necessarily the ISR component; ISR issues usually require 
“fleshing out” the scenarios. The DPG IPS are useful to size the force structure but 
does not necessarily cover totality of capabilities required 

e There is a need to model Red behavior in as much detail and fidelity as Blue 
behavior. 

e Who is the Decision Maker? In general, the SECDEF/Congress (e.g., QDR), but 
CINC requirements were included in subsequent discussions. Specific Decision 
maker is usually the DepSECDEF/VCICS. 
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Major Theater War: 
Current State of the Art (Summary) 


e Legacy Combat Models 
Scripted C2, some rule set behavior 
Some communications but simplistic 
Little effect shown from attacking C2 
Deterministic intelligence with little quality reference 


Limited explicit representation of C4ISR systems and mapping into 
combat outcome 


Sensor collection models may be adequate 
Intel and Combat models stovepiped 


Current State of the Art (Summary); (Slides 7 and 8): 


These two slides were prepared beforehand by the co-chairs. There was some discussion 
and disagreement over specific items on the slide, but the group agreed to the words that 
appear on this final version. Some specific comments: 

9“ Current (legacy) models aren’t all bad; modeling and analysis work to date have some 
useful attributes. However, in general, the state-of-the-art is woefully lacking in 
representation of C3 issues. ISR collection attributes are fairly well understood. Most 


importantly, the cause and effect of information on battle are not well understood or 
treated with existing models. 


e tis necessary to differentiate between a single model and a federation of models. The 
advantages and disadvantages of various approaches are discussed in later slides. 


9 Battle outcome as function of maneuver (e.g., flank attacks) are not well treated or 
understood. 


— The impact of information on C2 function is currently based on timeliness; the 
quality of the information available for changing decisions is not considered. 


— In legacy models, doctrine is hardwired. 
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Red C2 is needed to be reactive to Blue decisions, to show impact of Blue C4ISR. 
The impact may be dampened by red “inertia” in present analysis (i.e., Red 
follows same plans regardless of Blue capabilities). 

Communications physics is understood and can be modeled if desire is there. 
Communications are usually ignored because of the laborious and time 


consuming issues associated with the details of such efforts. The community has 
ignored the impact of these assumptions. 
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Major Theater War: 
Current State of the Art (Summary)-cont’d 


9 Campaign models under development - still no theory 
— Still much scripting 


— Using present models as basis, we still don’t have good C4ISR process 
representations in many areas 


-- Rule set behavior 
— Stochastic intelligence but simplistic fusion 
Mixed effect shown from attacking C2 


Major Theater War: 
State of the Art 


Collection Modeling 
IMINT --- Good 
ELINT --- Poor due to emissions not well modeled (scenarios) 
COMINT Scripted to Zero (emissions) 
Cross Cueing --- Poor 
MTI --- Marginal (Red Behavior) 
MASINT -- Non-imaging IR OK, the remainder non-existent 
HUMINT --- Humans???? 
PED Models 
--ὀ Good representation of time and volume flows 
— Poor representation of content (information vice data) 


— Dissemination and representation of organizational (hierarchy) 
knowledge poorly represented 


Fusion Models - non existent 


The State of the Art: 


The group decided that it would be useful to break up the ISR and C4 process in order to 
assess specific capabilities and deficiencies in the state-of-the-art representations of this 
process in current models. The following diagram was used: 


Collection 


The assessment was then made on various aspects of the C4ISR process within each box. 
In general, collection modeling was regarded as good overall (with specific deficiencies 
in some areas such as HUMINT, for example). PED models of total flow are mature but 
lack consideration of the quality of information, which is essential to answer the “80 
what?” response to volume measures. Fusion models are essentially non-existent, in real 
world as well as in analytic modeling. Obviously, some correlation and filtering is done 
in the real world but this automated process is regarded as immature. 
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Major Theater War: 
State of the Art (cont’d) 


Decision Modeling 
Rule set construction well understood 


What rules to use, especially in the future (different doctrine, 
CONOPS) not available 
Human factor issues not understood 
What decisions to model? Doctrine, “real world”, normative, optimal? 
Strike decisions / resource allocation relatively well understood, 
maneuver issues and effects not well understood 
Perception 

— COP representation does not reflect quality of information, the degree 

to which different sources agree/disagree, or alternative hypotheses 


— Ifthe COP has an area with no Red, is that because we have extensively 
observed the area and seen no Red, or is it because the area is obscured 
from view, or is it because we haven’t looked there? 


The State of the Art (Continued): 


Decision and perception modeling are also immature. Even if the process of representing 
a decision is known (e.g., rule-based inference), what decisions to use is up in the air. 
Unfortunately, the choice of decisions(s) to use in models often dominates the model 
outcomes, having an effect much greater than that realized by changes in systems or 
forces. Do we have the data to determine when decisions are made and how good they 
would be? Decisions should be evaluated as a function of risk based on the amount of 
information available, not necessarily the outcomes (hindsight). It is possible to make 
decisions that are “bad” from the view of the outcome but “good” with respect to the 
information available at the time that the decision was made. 
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Major Theater War: 
State of the Art (cont’d) 


* Communications 
Communications network modeling relatively well understood 
Physics well understood but not necessarily included 
Bandwidth and legal issues not represented 
Time and volume analysis moderately well understood 


Often don’t do the communication modeling we understand because of 
time and data constraints 


Often done off-line but this inhibits dynamics 


Difference between passing data (connectivity, delay) and passing 
information not understood 


Impact of loss of communications on C4ISR (which in turn effects 
battle outcomes) not well understood 


4 Planning / Re-planning 
— Plan execution and development relatively well represented 
— Re-planning poorly represented if at all 


The State of the Art (Continued): 


Communications modeling (physics based) is well understood; the modeling of 
information flows (vice bits per second) is not. Automated planning and re-planning 
capabilities are also lacking. ISR (collection) planning is easier because much is 
preplanned, but on the fly NAI’s are not done. 
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Major Theater War: 
State of the Art (cont’d) 


¢ Combat Outcomes 
— Strike relatively well understood 
¢ BDA and re-planning not as well understood 
— Maneuver not well understood 
¢ Even if we do C4ISR well, impact may not be reflected. E.g., what 
if we attack Red C4ISR and surprise him? What is the effect of 
surprise? 
Representing the effect of maneuver on Red behavior not 
understood (e.g., causing Red to withdraw without actually 
attacking him). 
— Total Force Protection and Focused Logistics are buzzwords with 
respect to C4ISR impacts 


¢ Limited functions (e.g., TMD) are reasonably well represented 


The State of the Art (Continued): 


Basic research is still needed in basic combat effects modeling. C4ISR capabilities should 
enhance maneuver, for example, but the effect of that maneuver is not well represented in 
current attrition-based models. There are two issues here. Our basic inability to tie Situational 
Awareness to a decision in a predictive manner means that we can not then link the decision 
to a battle outcome. The second issue is that most, if not all, ground combat models are 
insensitive to time and space of attack and what they mean to success. Therefore we can 
neither predict accurately what a commander would do differently as a result of information 
or show how that difference in movement or timing resulted in a change to his probability 
of success. As long as battle outcome metrics (casualties, force exchange ratios, etc.) are 
used to measure the effectiveness of new systems (to include C4ISR systems) and forces, 
better representations of basic combat effects will be required. Some specific observations: 
e Models are insensitive to outcome based on first order effects; this is much more 

acute in maneuver 
e There is little cross representation; e.g. Strike and IO impact on maneuver. 
e Legacy Models have good physics for movement and for some outcomes. 


The bottom line: 

e Campaign models are desired by decision makers -- MOFEs are preferred. 

e Itis necessary to move away from FER to representing maneuvering effects logistics, 
strategic effects, time to achieve objective, fratricide, IO, etc. 

e Current models all have significant/critical deficiencies for C4ISR analysis; some have 
partial capabilities 

e Major problems lie in combat modeling as well as in C4TSR models and integration 
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Other areas of deficiencies: 
Non-Combat 

Red C4ISR 

Red ROE decision heuristics 
IO 

Other Coalitions 


Major Theater War: 
Requirements for C41SR Analysts 


¢ Analysis Set Up 
Problem Definition 
Experimental Design 
Risk and Uncertainty 
Tools 
Scenarios 
Assumptions 


Sources 

Assumptions on data 
Aggregation 

Missing Data 
Incorporating Judgement 
VV&C 


Requirements for C4ISR Analysis: (slides 13 and 14) 


These slides are just meant to set the stage for those topics that we discuss in detail over 
the next 10 slides. 
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Major Theater War: 
Kequirements for C4ISR Analysis (cont'd) 


¢ Tools and Methodologies 
— Single campaign model (e.g., TACWAR, JWARS) 
— Hierarchy of Models 
— Federation of Models 
— Non-Simulation Tools 
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Major Theater War: 
Analysis Setup 


¢ Problem Definition 


More is Better? 

Defined Objectives 
Constraints? Casualties 
Time and Resources 


© Types of questions 


What is best among alternatives? 

How much better is A than B? 

How much A do I need? 

What is the tradeoff between A&B? 

How does A effect ....? 

How much does A cost? 

How well does A perform? 

Identify Offsets 

What is the marginal return of A in combat metrics? 
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Major Theater War: 
Problein Definition — Drivers 


Time 95 Security 


iderations 
Resources Cons 


~ People Coalition 
Considerations 


— Money 
— Available Data Availability 


Model Scenario Availability 


Question Culture 

Politics Previous Studies 
DM Expectations 

DM Focus 


Problem Definition: 


Some of the most critical issues surrounding studies of this magnitude have little to do 
with what may be called the classical design features of the scientific method. Instead, 
they revolve around completely exogenous factors that frequently end up determining the 
results out of hand. These variables include the time allowed to complete the effort, the 
resources provided, the politics of both the question to be answered and the organizations 
chosen to run it (as well as those left out), and the decision maker’s expectations 
surrounding results. These issues in a strict scientific sense should have no bearing on 
results and yet they frequently are the first order effects that determine the answer. 


Of the remaining issues the concepts of clearly articulating the question and working with 
the decision maker to insure that the question is properly understood by all and framed 
such that an answer can be derived is the most meaningful part of the analytical process. 
Scenarios, data and past work are all acknowledged as both required and unevenly 
provided and exercised in most study efforts. 
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Major Theater War: 
C4ISR MOES 


Definition of C4ISR MOPs and Battle MOEs not the problem.... It is the 
mapping of C4ISR MOPs to Battle MOEs 


Very Dependent on Context 
Analysis requires multiple MOEs 


MOEs: 


The definition of C4ISR MOPs is not a problem within the community from the 
perspective of the group. Nor is there difficulty in establishing criteria for mission 
success, e.g. MOEs. The real issues continue to be the ability of the analytical 
community to confidently establish direct linkages between the C4ISR system 
performance measures and the criteria for mission effectiveness. 


It is noted that no single MOP or MOE satisfies as a global metric for performance or 
effectiveness across all mission areas, nor is it sound analytical practice to only measure a 
single attribute as the criterion in a study effort. In studies as large and complex as those 
involved with MTW multiple criteria are required. 
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Major Theater War: 
Scenario and Excursions 


Deployment Timeline /Force Structure for C4ISR assets not represented 
Baseline Typically Directed | 
— Problems: 
° Multiple Detailed Versions of DPG Scenarios Exist 
* Multiple Predictions of Future, Conflict Areas 
° Little Definition of Red/Allied beyond initial lay down 
Should we vary Scenarios? How much? 
Red Scenario should be dynamic not fixed 


Scenarios and Excursions: | 


There are two issues embodied in this slide. The first issue is the fact that rarely, if ever, 
are C4ISR asset movements into theater accounted for. The second issue is that high level 
Scenarios such as the DPG do not specifically address the totality of the variables 
associated with setting up an analysis. 


The first issue results in unrealistic assumptions about C4ISR assets in theater early in a 
campaign and because of this likely overstate performance. The second issue requires the 
analyst to create portions of the scenario for a particular effort (e.g. specific force 
laydown, enemy behavior, own force ROE, etc.). All of these assumptions result in a 
multiplicity of scenarios eventually being created from what is believed by senior levels 
to be a homogenous baseline from which all efforts departed. 


Other concerns revolve around the issues of whether, in today’s environment, it is more 
applicable to analyze full blown MTW events rather than multiple Smaller Scale 
Regional Contingencies. 
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Major Theater War: 
Experimental Design 


Classical designs often not useful because statistical model not appropriate 


Too many things that should be varied - judgement on what needs to be 
varied 


Answers tend to be multidimensional; combination (via MAU?) often not well 
defined or explained to DM 


Exploratory analysis and prescreening may be useful 
Context dependent 


Experimental Design 


A review of recently completed analyses made in the context of an MTW shows that 
basic problems exist in terms of the sheer number of independent variables that can 
impact results. This “overflow of dimensionality” results in an experimental design that 
is unmanageable. The way recent efforts have dealt with the issue is to somewhat 
arbitrarily reduce the number of variables for which the analysis will study the impact. In 
so doing, the analytical team runs the risk of not completely exploring the cause and 
effect relationships and dependencies between multiple variables. The trade-off for this 
is obviously time and money. In general the group felt that more research needed to go 
into how to effectively manage this effort and maybe research that would provide insight 
into the variable dependencies that would allow constrained-run matrices in the future. 


Unfortunately, many analysts lack an appreciation for the statistical issues around the 
experimental design and interpretation of analysis result. For example, many of the 
packages and statistical techniques rely on assumptions such as normality and 
homoscedasticity, which are usually not tested for. Another example is the dearth of 
statistical tests for ordering means (vice determining if they are different). 
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Major Theater War: 
Data 


Few Accepted Sources for C4ISR Data 

Red C4ISR/Behavior data lacking 

Human Performance data lacking 

Hard to generate data for poorly understood models 


Future systems data lacking 


Existing models haven’t generated requirements for this type of data, 
therefore no one is tasked/funded to gather it 


Recording/cataloguing data from other analyses is necessary 


Data: 


The discussion here centered on the distinctions between system performance data, 
Scenario data, and “behavioral” data (e.g. human decision making). Although there are 
always issues of how to characterize future systems and difficulties in getting some 
organizations (e.g., NRO) to share data, the group felt that MTW was the most mature 
area of data collection and storing in the community. There are several organizations that 
are chartered to collect and hold such information for general use and it is also the most 
easily understood data within the non-modeling community. The next layer of data, 
Scenario data, has many problems associated with it. As alluded to above, multiple 
analyses seem to beget multiple interpretations of proscribed “baselines”. In addition this 
type of data is incomplete in its representation of red forces especially in terms of Red 
C4ISR systems and Red Behavioral data. The final and most vexing area of data required 
is that on behavioral entities. There are virtually no reliable data sources for C2 events 
that require decision making to occur or data that suggests what courses of action may 
transpire from equivalent battlefield events (Red or Blue). The group felt that in order to 
advance the science of C4ISR modeling much research was required here, followed by 
the nomination of a group or organization to manage and store the results. 
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Major Theater War: 
Data (Cortinued) 


Existing data doesn’t necessarily map into DPG scenarios 
Big difference between physical data and behavioral data 


Background Data (civilian traffic, etc.) not available “Baseline” Data hard to 
define when C4ISR changes can change scenario, decisions, behaviors 


Scenario Data developed for unique modeling constructs 
Security Issues 
Politics/Data Sharing 
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Major Theater War: 
Methodology/Tools 


Monolithic Single Campaign Model 


Effective Use: Problems: 
Calculates MOFE 95 Theoretically Flawed 
Full context of conflict 95 Tend to focus on attrition 
Demonstrate what you ° Very complex and data intensive 
already decided We are building a tool based on 
OP Plan Analysis today's understanding of C4ISR 


Combine with higher-level impacts, while no one really 


models to explore particular knows the true mechanisms by 
issues which C4ISR impacts battle 


V&V with evolving models 


Models and Tools: 


The group examined four major approaches to representing C4ISR systems in the context 
of an MTW campaign analysis. These approaches were: 

e Developing a single, monolithic campaign model that represents “everything,” 

e Developing a hierarchy of models, 

e Developing a federation of models, and 

e Using nontraditional techniques outside of simulation. 


Pros and cons of each approach were discussed. 


Monolithic: There are two major points that need to be brought out when looking at large 
campaign level monolithic models. The first is that they are by their nature low fidelity 
representations of the combat and combat interactions, as such it is extremely hard to 
identify cause and effect relationships for those features that are not explicitly designed 
into the model as a major driver of combat results. Given that this has never been done 
for C4ISR it is next to impossible to use such models today to adequately address the 
impact of information. The second issue is that it will only be possible in the future if the 


theory of C4ISR impact on battle operations is canonized and put into these models as a 
first order driving effect. 
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Another observation not well appreciated by senior decision makers is the fact 
that current C4ISR-based advanced modeling efforts (e.g., FWARS and JSIMS) will not 
provide any better representation of C4ISR systems and processes that the current (poor) 
state-of-the-art allows unless research (vice coding) is invested in. 
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Major Theater War: 
Methodology/Tools (Continued) 


Hierarchy of Models 


Effective Use: Problems: 


Data generation for low-res ° Aggregation/Dissaggregation 
models ° Model treatments often 
Determine what issues to inconsistent 

look at Confidence Intervals 

“Flesh Out’ detailed issues .--Accuracy measures across 
Cross-check “validity” multiple models 


Depth and Breadth Choosing what to represent at 
higher resolution 


Multiple answers to DM 


Hierarchy: 


This approach has one main draw back. Data is usually modeled at different levels of 
fidelity across the continuum of the modeling structure. Whether it is the data used to 
created the scenario or the data used in treating impacts of various attributes of the war 


they are probably considered in differing degrees of fidelity and therefore the linkages of 
models becomes suspect. 
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Major Theater War: 
Methodology/Tools (Centinued) 


Federation of Models 


Effective Use: 
© Accepted Models do Exist ¢ Target Domain mapping 


Problems: 


9 Treats Individual Functions in 9 Aggregation/Dissaggregation 
Higher Fidelity than Campaign e 


Time and Space Discontinuities 
model allows 


Φ Model treatments inconsistent 
ὁ Insures Cause and Effect 


Explicitly Treated and © Feedback loops lacking 


Understood ¢ Hard to ID Synergy 
¢ Data generation for Combat * Missing Pieces 

Models ¢ Multiple answers to DM 
e “Flesh Out” detailed issues . VV&C 
5“ Cross-check “validity” e Abuser Friendly 


Depth and Breadth 


Federation: 


To date this has been the most successful method for capturing the cause and effect 
relationships implicit in C4ISR on the battlefield. It allows fidelity where fidelity is 
needed and allows the analytical team to review the cause and effect relationships as they 
unfold at the seams of the individual models. These types of modeling efforts still fall 
prey to the same issues discussed above for Hierarchy of models. 
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Major Theater War: 
Methodology/Tools (Continued) 


Non-Simulation Approaches: 


¢ Spread Sheets 
° Analytic Model 
° Man-in-The-Loop 
War games 
Exercises 
Historical Analysis 
Surveys/Operator Observations 


Non-Traditional Techniques: 


A variety of non-simulation approaches may be used to answer parts of the analysis 
question (although it did not appear that any single technique was robust enough to 
approach answering all C4ISR-related questions in an MTW context). These techniques 
are listed on the slide, but should be considered to be a representative sample of the 
techniques available rather than a comprehensive list. 
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Major Theater War: 
Assessment of Current State-of-the-Practice 


Assessment of the State-of-the-Practice: Using the same chart as used by other working 
groups: 


Problem Structuring (Yellow): 

e Strike issues relatively well understood, maneuver issues poorly understood 
e Link between C4ISR MOEs and MOFEs not well understood 

e System level performance issues & MOEs relatively well understood 


Human Factors and Organization (Red) 

Behavior models not developed 

No Red behavior models, data, or dynamic scenarios available 

Data for behavior and decision making not available 

Human-in-the-Loop approaches are not repeatable and may not be verifiable 
Do we model doctrine, “real-world,” normative or optimal decisions? 


Decisions tend to cause significantly nonmonotonic outcomes. They also tend to be 
nonlinear with respect to the stimulus (e.g., thresholds) 


Assessment of the State-of-the-Practice (cont’d): 


Scenarios (Red) 


e DPG scenarios really don’t get at C4ISR issues 


e Conflict between a defined (scripted) scenario and showing C4ISR effects (which cause dynamic 
effects) 
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Red behaviors (especially dynamics) not defined in scenarios or supporting data 
For ISR, DPG / MTW scenarios may not be the most stressful condition (DCI/peacetime may be 
largest/most stressing requirements) 

Issues that have great impact on C4ISR may not be significant in the scenario (e.g., SCUDs that 
launch against political vice military targets) 

Decisions & doctrine not defined in future scenarios 


Measures of Merit (Yellow) 


C4ISR MOEs and MOFEs well understood but linkage between them is not 
MOMs very dependent on context — hard to make any general statements 


A good decision is not synonymous with a good outcome. This creates a problem when you use 
MOFEs as criteria for evaluation 


Tools and their Applications (Red) 


No good models or tools exist for many C4ISR processes 

Red C4ISR not well understood or modeled 

Models under development (e.g., JWARS) are taking the “best current modeling approach” as their 
basis — since no good models exist currently, JWARS will not have any models to incorporate. 
Hierarchical and federated modeling approaches have significant problems with model consistency 
You can’t model well what you don’t understand 


Data (Red) 


Data generally does not exist on behavior and human issues 

Red C4ISR data particularly lacking (especially behavior) 

Doctrine is a form of data for C4ISR models — yet we don’t have that well quantified, especially 
when future systems are included 


There is a big difference between physical attribute data and behavioral / decision data 
Funding is not there 


Risk and Uncertainty 


Will the systems perform as expected? Relatively well understood 


e Will the systems effect combat as modeled / hypothesized? Not understood 

e Is the context correct? Not well defined for C4ISR — e.g., doctrine, future baseline, data, etc. 

e Uncertainty is often modeled with probability— but we don’t understand the models, much less the 
distributions, much less the distributional parameters 

Report (Red) 


Because we lack C4ISR models and data are lacking, many assumptions have to be made. These 
assumptions (and their impact) may not be well reported to and understood by the DM 


Linkage assumptions between C4ISR and combat outcomes often not reported (may not even be 
understood by analyst if implicit in models) 
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Major Theater War: 
Current State of the Art (Detail) 


Breakout of the state-of-the-art within the intelligence cycle: 


These ratings were discussed previously; this slide simply summarizes the ratings. 
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Major Theater War: 
Recommendations 


Research needed to how we reflect C4ISR outcomes into military effects (e.g., 
maneuver) 


Research needed on perception development and fusion 

Research and data needed on decision making (C2) representation 
Funding needed for data collection and management. Cannot just be tied to 
existing models. (Will save analysis money in long run) 

We need to explain better to DM where model and analysis limitations are. 
Analysis demands seem to have shorter timelines and less money. “Quick 
turnaround” analysis may require extensive preparation (scenarios, data, 
etc.) be accomplished beforehand. 


New model development (e.g., J WARS) is based on “using the best of the 
accepted algorithms.” Since C4ISR algorithms are missing now, the next 
generation of models will not have proper C4ISR representations. 
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Major Theater War: 
Recommendations (Cont’d) 


Need to develop knowledge base for Red behavior (not just systems and 
general doctrine). Coalition behavior information also needed. 


JDS needs specific tasking to support C4ISR analysis beyond JWARS. This 
will require additional funding. 


Intel scenarios and combat scenarios need to be matched. Non-tactical intel 
requirements need to be defined in context of campaign. 


To ensure consistency across services, the QDR scenarios need to be 
fleshed out now by (or managed by) a single joint agency. This includes 
data and Red representation. 


More data sharing and sanitization of data (e.g., national systems) needed. 
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Major Theater War: 
The Bottom Line 


° Insight: Basic research is needed. Some major areas (not all inclusive): 
— Linkage between C4ISR effects and campaign outcomes, 

— Fusion, perception development and decision making, 

— Basic combat itself (e.g., maneuver effects). 


° Insight: JWARS and other new model efforts are incorporating the “best” 
current models - this approach will not work with respect to C4ISR. 


* Recommendation: Develop prioritized list in the areas of C4ISR effects on 
combat, C2, ISR, communications and basic combat to direct the research 
plan. Action: J-2, J-6, J-8, DSC (J-6 lead). Fund research, Action: 
DD(R&E) / ASD(C3I). This research needs to be implemented into new 


model developments, but we do not need the effort focused on a particular 
tool. 


Bottom Line” Recommendations: 


The “Bottom Line” slides include the Working Group’s recommendations regarding the 


appropriate implementing organizations. Those recommendations have not been formally 
coordinated with the organizations listed. 


Major Theater War: 
The Bottom Line (Cont’d) 


Insight: There are fundamental data gaps to support C4ISR. 


Recommendation: 
Enumerate the data requirements (more than JWARS support), 
Action: ASD(C31/DSC/Joint Staff/IC 
Populate the data bases, Action: ASD(C3I) DSC/JDS/DD(R&E) 
DMSO 
Make available to community. Action: JDS 


Insight: C4ISR analysis will require a range of different models, which may 
include but will not be limited to JWARS. 


Recommendation: Develop a requirement and funding plan to develop the 
analysis infrastructure (data, tools, etc) needed to respond to current 
demands. This is more than JWARS and JDS. This needs to be in place by the 
QDR. Action: J-2, J-6, J-8, DSC (J-6 lead). 
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MILITARY OPERATIONS RESEARCH SOCIETY 


Analyzing C4ISR for 2010 


Smaller Scale Contingencies 


Smaller Scale Contingencies (SSC) proved difficult to define well and difficult to 
differentiate from Operations Other Than War (OOTW). The group chose to focus its 
efforts on mission types in which the use of military force was likely (raids, hostage 
rescue, opposed NEOS, etc.), which were also discussed as those for which military 
considerations tend to take precedence over political considerations. This distinction was 
adopted in order to focus the Working Group on a different arena from the OOTW group, 
which focused most of its energy on Peace Operations and Humanitarian Assistance and 
Disaster Relief missions. 


The group recognized that this is an artificial distinction. Many formal definitions of 
OOTW explicitly include SSC. Moreover, SSC intended as nearly pure military 
operations often are changed into more political activities. The scenario played out in 
Haiti (which went in a matter of a few minutes from an invasion of a small country to a 
friendly occupation with primarily political goals) provided a good example of this 
phenomenon. Finally, as the group stressed in its deliberations, SSC tend to have 
significant political dimensions, even when the operations themselves are largely military 
in character. 


The Working Group also concluded that analysis of SSC cannot be meaningfully 
conducted on the group of missions as a whole. Their diversity requires that tools and 
analytic efforts be focused on meaningful, coherent subsets of the topic. For example, 
studies of raids will require different tools and data than studies of hostage rescues. 
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Smaller Scale Contingencies: 
Participants List 


NAME 
Anway-Wiese, Carol 


Cann, David 


Chartier, Chris 

Cherolis, George 

Coe, Gary 

Elton, Olaf 

Forsythe, Steven L. (Major USAF) 
Hayes, Richard E. (Chair) 
Herlihy, Anthony R. (Major, 
USMC) 

Kirkland, Joseph K. 

Macklin, Marilyn 

McGregor, Otis (Captain, USA) 
Morrell, Angela 


Oswalt, Ivar 

Peterson, Pat 

Richards, Dale 

Siegel, Adam 

Wheatley, Gary (RADM USN Ret.) 


Who Participated: 


ORGANIZATION 
Boeing 
Naval Undersea Warfare Center, 
Newport 
OASD (C3D/Decision Support Center 
TRW Tech Support for TACCSF 
IDA 
MITRE 
AFSAA/SAAB 
Evidence Based Research Inc. 
Marine Security Guard Bn. 


Evidence Based Research Inc. 

HQ, Dept. of the Army 

US Army LIWA 

US Special Operations Command 
SOIO-C4I 

Kapos Associates Inc./N6M Support 
SAIC 

USAF Research Laboratory/IFTB 
Center for Naval Analyses (CNA) 
Evidence Based Research Inc. 


The eighteen individuals who participated are listed on the slide. They represented an 
excellent mix of C4ISR analysts and operators with widely divergent backgrounds and 
training. While this diversity made communication difficult at times, the members of the 
Working Group did an excellent job of working with one another. No one perspective 
dominated and a remarkably coherent and consistent perspective emerged over time. 
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Smaller Scale Contingencies: 
Characterize C4ISR 


9 Require more rapid & agile: 
— Intelligence processes 
— Continuous/dynamic planning & execution 


e Pressure toward centralized control 
— “Only game in town” 


— Strategic implications of tactical activity 


9“ Danger of different understandings of the situation across command echelons 


Characterize C4ISR (1): 


While the Working Group was cognizant of the need to differentiate SSC from other types 
of operations, many of the ways that C4ISR needs are characterized in this arena would be 
the same as those in other arenas. For example, looking at future SSC, the dominant 
requirement will be for more rapid and agile intelligence, planning and execution processes. 
While not unique to SSC, this need will be important and analysts must be looking for ways 
to ensure it is met. This means exploring ways to improve the speed and responsiveness of 
the entire ISR process (from tasking through dissemination) as well as to couple situation 
awareness, decision making, and execution more closely. 


SSC are different from MTW, at least, in that they almost always involve considerable 
pressure for centralized control. This arises both from the legitimate concerns of higher 
command arising from the potentially strategic implications of military activities down to 
the 

tactical level and the more idiosyncratic fact that SSC are often "the only game in town," 
which means that decision makers at all levels are free to invest time in them. MTW, on 

the other hand, are often large and complex enough that senior political and military officials 
often are to busy to focus on tactical level decisions and actions. 


Given the high level of attention likely to be generated, the danger also arises of different 


understandings of the military and political situations at different levels of command. These 
disconnects can threaten the coherence and ultimate success of the operation. 
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SSC may (but do not always) also require a high level of tactical security (particularly in the 
planning, preparation and entry phases) and management of highly visible consequences. 
The experience of conducting an operation that is a military success yet fails in terms of US 
policy was all too familiar to the Working Group participants. 


Communication issues arose several times during the discussion. SSC often require new 
or novel infrastructures that overcome gaps arising from military service differences, the 
need to include coalition partners or host nations, or the need to deal with non-military 


actors (inter-agency or NGO). Moreover, secure communications may be needed to achieve 
adequate operational security. 


Smaller Scale Contingencies: 
Characterize C4ISR (contd) 


e May Require: 
— High level of tactical security 
— Management of highly visible consequences 
— Specialized communication capabilities: 
9 Infrastructure 
e §6Security 
94 Coalition 


Smaller Scale Contingencies: 
Key Issues and Questions 


SSC not necessarily a lesser included case of MTW 
— MOFE and MOPE not “controlled” by the military 


— Intelligence required tend to be more fine grained and detailed than for 
MTW 


Definition of mission success/failure often includes constraints not present in 
MTW. 


Tension between: 
— Avoiding mission failure 
— Maximizing likelihood of mission success 


Key Issues and Questions: 


The Working Group wanted to emphasize that SSC are not necessarily a lesser-included 
case of MTW. This is crucial when data is being selected, models chosen, parameters 
estimated, or research is being organized. Some of the distinctions were mentioned 
earlier. Others include the fact that the MOFE and MOPE are usually not under the 
control of the military. In this sense, SSC are properly understood to be OOTW. 
Moreover, the group, and particularly the operators in the group, stressed the fact that the 
intelligence required in many SSC is more fine grained and detailed than that required for 
the same levels of command in MTW. 


SSC are often conducted under restraints that would not be present in MTW. Rules of 
Engagement (ROE) and political restrictions often limit the activities available for 
mission accomplishment. Moreover, SSC tend to be characterized by a very meaningful 
tension between efforts to avoid mission failure and maximizing the likelihood of mission 
success. Examples of decisions to limit the size and armament involved in operations in 


order to avoid undesirable political consequences were offered by several members of the 
Working Group. 
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Smaller Scale Contingencies: 
Relative Worth 


Improved C4ISR in SSC should yield benefits such as: 
— Economy of force 
Agility and synchronization 
Improved operational performance 
Improved policy effectiveness 
Improved mission accomplishment 


Relative Worth: 


The group noted that improved C4ISR should pay off in some very specific ways in SSC. 
In particular, greater agility in situation awareness, planning and execution were placed 
paramount in importance. Not only better operational performance, but also improved 
likelihood of mission accomplishment and policy effectiveness were considered possible. 
Additional benefits should include improved agility and synchronization and greater 
economy of force. 


Smaller Scale Contingencies: 
Assessing Relative Worth 


° Assessment of the relative worth of C4ISR systems requires considering: 
— Range of SSC missions and their relative likelihood 
— Impact on MOFE 


5 Tools and methods for assessing relative worth require substantial 
verification, validation, and accreditation (VV&A) 


Assessing Relative Worth: 


The method for assessing the relative worth of C4ISR systems was seen as readily 
available from fundamental operations research techniques. This is not to say that the 
data and tools needed to conduct those analyses are available. Simply stated the group 
recognized the value of a particular C4ISR innovation could be seen as a function of the 
range of SSC missions to which it applies, the relative likelihood of those mission types 
and their estimated impact on the relevant MOFE and MOPE. After explicit discussion, 
the group rejected the idea that the "importance" of the mission type be added to the 
equation, arguing that any one type of mission might be crucial to US national interests 
under selected circumstances. The obvious example of this is the crucial role of the 
blockade during the Cuban Missile Crisis. 


The working group also felt compelled to point out the need for verification, validation 
and accreditation (VV&A) for tools, methods, and data being used to conduct these 
relative worth analyses. 
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Smaller Scale Contingencies: 
Measures of Merit 


“Dimensional » 
Parameters 


Measures of Merit (1 and 2): 


The group readily accepted both the set of measures of merit categories defined by 
MORS in the 1980s and the suggested change offered by Dr. Stuart Starr during his 
plenary address, of adding an outside layer, Measures of Policy Effectiveness (MOPE). 
Equally important, the group also wanted to ensure that analyses of C4ISR in SSC "be 
informed by" metrics from all the relevant layers. The conclusion was not that all 
analyses include all levels of measurement. For example, many analyses are focused on 
existing equipment for which the dimensional parameters are fixed. However, the 
Working Group wanted to emphasize the need for analysts to be aware of the limits of 
analyses that do not consider all the levels. 


Smaller Scale Contingencies: 
Measures of Merit (cont’d) 


Analysis of future C4ISR systems should be informed by measures of merit at 
all the MORS-defined levels 


Interoperability 
Distribution of information: 


— Include military, interagency, and coalition partners 
— Appropriate balance of push/pull 


Measures of Merit (1 and 2, cont’d): 


Interoperability issues also arose several times. Despite the emphasis on this topic since 
Grenada, it continues to bedevil operators in SSC. This also confounds (but is not the 
only confounding factor) the distribution of information. Despite changes in policy 
arising from the Bosnia experience, the people on the spot who do the planning and 
support the execution of SSC are often unaware of, or are unable to acquire ISR products 
in a timely manner. This problem arises across service, national and functional lines. The 
correct balance of information push and information pull has not yet been achieved. 
Metrics for interoperability and for information dissemination remains important in 
assessing future C4ISR systems. 
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Smaller Scale Contingencies: 
Measures of Merit (cont'd) 


® Quality of information is the key dimension 
— Responsive to war fighters 
¢ Cover CCIR 
9 In useable format 


Characterized as: 

Complete 

Current 

Correct 

Precise enough for purposes 

Consistent: internally and across command centers 


Measures of Merit (3): 


The quality of the information available was seen as the single most important metric for 
SSC. This was defined to include its responsiveness to warfighters, including the 
Commander's Critical Information Requirements (CCIR), but also the needs of key staff 
members and the units executing the operation. Responsiveness was seen as including 
much more than mere availability. Some operators reported "drinking from a fire hose" 
when they were given "all the relevant" material. Others noted that they often received 
material in a format that either was not readable or was difficult to use. Here, again, the 
proper balance of push and pull is crucial. 


What is meant by high quality information was based on the Headquarters Effectiveness 
Assessment Tool, which was familiar to several members of the Working Group. 
"Perfect" information in a military situation is seen as complete (for the command's 
needs), current, correct, precise enough for the associated military purposes (which will 
vary from targeting, which is very precise, to general maneuver planning, which is often 
much less precise) and consistent. Consistent here means both internally consistent 
within the data, information, and knowledge of the C4ISR system and consistent across 
command nodes. 


Smaller Scale Contingencies: 
Measures of Merit (cont’d) 


¢ Anticipation of Key Demands for: 
— Intelligence 
— Logistics 
— Operations 
— Communications 
5. Rapid and robust situation assessment, decision making, planning, execution: 
— Speed 
— Variety of alternatives 
— Quality (effectiveness) 


Measures of Merit (4): 


The group also wanted to stress the need for future C4ISR systems that will be 
anticipatory across the full range of key demands that arise in command and control: 
operational, intelligence, logistics and communications. In other words, if continuous 


and effective planning and execution systems are to work, the system must be thoroughly 
cross-linked in very near real time. 


Similarly, measures of merit that focus on rapid and robust decision making, planning 
and execution will be needed. This means we need valid and reliable indicators of the 
speed of the C4ISR processes, the variety of futures and alternative courses of action they 
can and do consider, and the quality of the decisions they make. Indicators of decision 
quality essentially must be MOFE and MOPE, though MOE for C4ISR, MOP for C4ISR - 
systems and the dimensional parameters that drive them may also be needed for 
diagnostic purposes in any given analysis. 


Smaller Scale Contingencies: 
Tools Needed 


9 Improved support tools for COA 
— Credible projections of alternative futures 
— Capacity for “what if?” analysis 
— Fast enough to support mission planning 
— Miust be able to pull from current operational databases 


4 Virtual C4ISR rehearsal capability 
— Analytic tool 
— Operational tool 


Tools Needed (1): 


This area was perhaps the least mature of the areas discussed in the working group. 
Tension existed between the desire to identify tools for C4ISR analysts and tools needed 
by the operators. The requirements were seen to be quite similar in many cases and the 
group identified several tools that would be of value to both. 


The most important tool requirement was for credible support to course of action (COA) 
analysis. This means tools that generate projections of alternative futures that are 
credible to operators and well as the capacity for sensitivity or “what if" analyses. These 
tools must also be fast enough to support mission planning and have an analytic mode 
that permits large numbers of runs in a short period of time. Moreover, these tools must 
be designed to pull from operational databases, not just notional or analytic ones. 


Perhaps the most novel idea arising for new tools was a call for a virtual C4ISR rehearsal 
capability that would be designed both as a tool for analysts and for operators. Operators 
would use it for training and mission rehearsal. Analysts could use it for improving their 
understanding of missions, generating hypotheses about C4ISR improvements, 
examining man in the loop elements, and testing hypotheses. 
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Smaller Scale Contingencies: 
fools Needed (cont’d) 


5 Improve data collection and contro! 
° Identify and link simulated and real data 
5 Future systems better integrated across: 
Functions 
Echelons of Command 
Time 


Capability to model/ameliorate the impact of language, culture and 
training differences 


Tools Needed (2): 


The group heard a similar plea for more effective ways to both link and differentiate real 


and simulated data, particularly in real headquarters participating in virtual exercises and 
training. 


In general, tools are needed to assess the degree to which C4ISR systems are linked 
across functions, echelons and time. New information systems are linking operators more 
and more closely, but no tools to measure these changes or their impacts are currently 
available. Similarly, models that capture the effects of "soft factors" such as differences 
in language, culture, or training are not available but would be valuable in an era that 
stresses "jointness” and coalition operations. 
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Smaller Scale Contingencies: 
Tools Needed (cont’d) 


Virtual C4ISR rehearsal capability 
— Analytic tool 
— Operational tool 


Simulation for rapid integration and deconfliction of communications 
Rich relational databases with improved linkage software 
Uncertainty estimator and communicator 


Ability to capture and replay data/voice/imagery traffic for current and 
future operations 


Tools Needed (3): 


The number and variety of communications “horror stories” called out by both the 
plenary speakers and the operators in the Working Group were seen as indicating a need 
for a simulation that would focus on the capability to rapidly integrate and deconflict the 
"kluges" of systems often required in SSC. 


Rich relational databases, focused on the SSC mission areas were also seen as important, 
as was development of improved linkage software to speed searches and enhance their 
utility. Finally, systems that can estimate and communicate uncertainty in the context of 
C4ISR systems and the information they present were also seen as important. 


Analysts, particularly those supporting operational organizations, also felt a need to 
capture and replay information (data, voice traffic, imagery, etc.) for current and future 
operations. This would greatly facilitate after action analyses from both operations and 
exercises. 
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Smaller Scale Contingencies: 
Examining Illustrative C4ISR Smaller Scale 
Contingencies 


Mission | Future of Analytic Measures Key Metrics 
Types C4ISR Tools of Merit 


> Predictive Mission Accom- 
Modeling plishment 


> Virtual Collateral 
Rehearsal Damage 


Interoperability 
Speed 


Targets 
Destroyed 


Minimum US 
Casualties 


Illustrative Examples (1) — Raid: 


Slides 15 through 18 represent detailed work performed by the working group, which 
actually examined four different types of SSC mission areas to link future C4ISR needs, 
the analytical tools required to support their development and refinement, the most 
crucial measures of merit, and the implied key metrics. 


In the Raid analysis the group determined that the critical C4ISR product is timely 3D 
Imagery. This imagery is essential to support the needed analytic tools of Predictive 
Modeling and Virtual Rehearsal, which were in turn believed to offer potential for greatly 
improved mission success. Measures of merit are dominated by MOFE and MOPE, with 
Force Effectiveness deemed somewhat more important the Measure of Policy 
Effectiveness. Key Metrics include Minimum US Casualties, Minimum Collateral 
Damage, Interoperability of C3I and number of Targets Destroyed in the raid. 
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Smaller Scale Contingencies: 
Examining Illustrative C4ISR Smailer Scale 
Contingencies (cont’d) 


Future of C4ISR Analytic Tools Measures of Merit 


Global Positioning Predictive Inter-operability Minimum US 
System Modeling Casualties 


Databases as Intel Virtual Economy of Collateral 
Products Rehearsal Force/Cost Damage 
Effective-ness 


Communications Maneuver Maneuver- Interoperability 

Continuity Models ability of communi- 
cation and 
computers 


Identification Intel/Ops "Key Nodes to 
Friend or Foe for Automation Take" in urban 
land forces terrain 
Language Communi- 
Translators cations 

simulation 


Illustrative Examples(2) -- Urban Warfare: 


Urban Warfare offers perhaps the biggest C4ISR challenges today and in the future. 
These include enhanced utilization and exploitation of GPS for precise three dimensional 
positioning, Combat Identification of friend and foe in densely populated areas where 
non-combatants are often mixed with the enemy, and communications that will function 
seamlessly from sewers to skyscrapers. Detailed databases are essential to support 
Predictive Modeling and Virtual Rehearsal tools. Measures of Merit stress 
Interoperability and Economy of Force with key metrics similar to those in the previous 
example. 


Smaller Scale Contingencies: 
Examining Illustrative C4ISR Smaller Scale 
Contingencies (cont’d) 


Mission Types 1 Future of Analytic Tools Measures of Key Metrics 
C4ISR Merit 


Counter- Relational > Highvolume > Hits verses » Decrease 
terrorism/ databases of data false alarms in 
Operational with linkage terrorist 
Disruption software attacks 


Better Knowledge- ΣῈ Level of > Increase 
organized able domain confidence in 
data experts terrorist 
threat 

Threshold > Prioriti- > Level of 
control zation uncertainty 

“tags” for 

intelligence 

alerts 


Illustrative Examples(3) -- Counter Terrorism/Operational Disruption: 


Terrorism and other trans-national threats are quickly becoming epidemic. As developed 
in previous counter-narcotics analyses, terrorists, drug czars and similar actors can be 
identified and located through concentrated research and analysis. The key is the data, 
and future C4ISR to address these threats will require much better organized data that can 
be linked through relational databases. Analytic tools can then use the data to identify 
and locate terrorists. Measures of Merit will address the success of the analyses and 
focus on accuracy of hits versus false alarms, and the confidence level of the alerts. 
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Smaller Scale Contingencies: 
Examining Hlustrative C4ISR Smaller Scale Contingencies 
(cont’d) 


Mission Types Future of C4ISR_ Analytic Tools Measures of Καον Metrics 
Merit 


Opposed NEO/ | > Information > Modeling& Σ» MOFE< > Mission 
Potentially on terrain Simulation MOPE Accomplish- 
Opposed NEO Anticipatory ment 


Improved COA Quality of Minimum US 
interagency Analysis informatio and civilian 
/ coalition Tool n casualties 
C4ISR 


Ability to MOP Increase in 
monitor capability of 
situation number of 
and people 
operative evacuated 
environment 

Detailed 

Intel 

products 


Illustrative Examples(4) -- Opposed/Potentially Opposed NEO: 


An Opposed or Potentially Opposed NEO is perhaps one of the most difficult and 
delicate missions which our armed forces are called upon to conduct. These missions 
literally mean life or death to US and other friendly civilians, and create high visibility 
political consequences. Key C4ISR capabilities needed are detailed and current 
intelligence products to include precise terrain and building data, and continuous or near 
continuous location information about evacuees and potential opponents. M&S tools that 
provide rehearsal capability and COA analyses are particularly needed. In this mission 
MOPE and MOFE are paramount with MOPE most important. Mission accomplishment 
with minimum US and civilian casualties are the Key Metrics. 
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Smaller Scale Contingencies: 
ssessment of State-of-the-Practice 


SERRE ENE RN MA γ meen! 


Assessment: 


This slide is self-explanatory except that the Working Group felt that many areas fell 
between red and yellow and chose to color them orange. 
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Smaller Scale Contingencies: 
Recommendations 


Develop clearer definitions to support analysis 
Focus on specific missions rather than artificial categories 


Baselines are essential for good analyses and must be developed for the 
mission types included in SSC 


Invest in the C4ISR analysis tools identified by the working group (e.g., 
VV&A) 


Recommendations (1): 


The recommendations were difficult to draft and represent only a fraction of the rich 
discussion and debate of the working group. Those recommendations ranged from: 


e the difficulty of defining SSC (ultimately resolved in the very practical way of 
noting that the included mission types are really the focus of analysis) 
the argument that baselines do not exist for most of these mission types 
meaningful analysis must have those crucial baselines for assessment of the 
contribution of current and future C4ISR systems. 


The most fundamental recommendation was that DoD and the MORS community begin 
an investment program in the tools identified as necessary. 
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Smaller Scale Contingencies: 
Recommendations (cont'd) 


Further research to develop methods for collection and analysis of the 
importance of measures of policy effectiveness (MOPE) for SSC 


Invest in better storage and distribution databases of relevant C4ISR 
information for each SSC mission area 


C4ISR must be interoperable, including interagency and coalition players 


Recommendations (2): 


The area of MOPEs was also called out as an area crucial to further progress in SSC. 
This finding was expected to be similar to one reported by the OOTW Working Group. 
Investment in databases (and information and knowledge bases) to support SSC mission 
area analyses was also identified as a very important recommendation. 


Smaller Scale Contingencies: 
Summary 


Future and current needs of C4ISR within SSC must be mission focused 
SSC require more detailed, “fine grained” data from C4ISR elements 


Investment in relevant C4ISR information and C4ISR COA tools 
recommended 


Development of baselines, information and further research necessary 


Summary: 


The current and future C4ISR needs for Smaller Scale Contingencies must be addressed 
within the context of each specific mission category, as opposed to addressing a broad 
range of similar missions. By heeding this, proper analysis of each mission area may be 
secured. C4ISR data within Smaller Scale Contingencies will need to develop a higher 
level of tactical detail and control then other arenas of warfare (i.e. peacekeeping, peace 
enforcement, major theater war.) This characteristic, developed from tight operational 
constraints, will aid commanders' tactical planning, control and security. Investment in 
such C4ISR information systems such as virtual rehearsal programs, course of action. 
analytical tools and relational databases is recommended to increase military force 
flexibility and performance. 
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ΠΙΌ ΕΣ 


MILITARY OPERATIONS RESEARCH SOCIETY 


Analyzing C4ISR for 2010 


Operations Other Than War 
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Operations Other Than War: 
Working Group Taskings 


Characterize C4ISR within WG focus area 
— Decompose the elements of C4ISR 


Define the relative worth of C4ISR 
— C4ISR/Firepower/Force Employment 


Develop & Recommend Measures of Merit 
- MOPs and MOEs 
— How well they value C4ISR, estimate ROI 


Identify and describe tools 
Analyze, synthesize, and infer 


Working Group Tasking: 


The Terms of Reference charged the Working Group to undertake five tasks: 


I. 


Across the range of OOTW missions, decompose the aspects of C4ISR and consider 
the analysis issues associated with them. The objective is to consider the value of 
CISR contributions. 


. Continue the assessment by considering the relative worth of C4ISR to “firepower” 


and force employment. Note here that “firepower” is a misnomer for force 
capabilities and systems, since generally the missions and tasks of OOTW will 
require other than the prime warfare systems. 

Considering the types of analysis to be performed, develop measures of merit (MOPs 
and MOEs) that are appropriate to the mission areas. Also, address the ability to 
conduct assessments of return on investment. 

Identify tools and methods that are necessary to the analysis described above. 

Finally, consider all the information generated to draw conclusions and observations. 


Operations Other Than War: 
Group Participants 


Art Dougas Booz-Allen 

John Furman Mitre 

Bob Hartling OPNAV 81 

John Kelsey TASC 

Randall (Mel) Parish TRAC-WSMR 

Terry Prosser Logicon 

Mark Sinclair Joint C4ISR Battle Center (Veridian) 
Mike Sovereign Naval Postgraduate School 

Cy Staniec Logicon 

Don Theune Virtual Technology Corporation 
Corinne Wallshein AFSAA/SAAB 

Scott Welch Evidence Based Research 

Russ Richards Mitre (Synthesis Group) 


OOTW Group Participants: 


Dr Cy Staniec chaired the group, with John Furman and Terry Prosser as co-chairs. The 
group included a broad array of skills and experiences, including recent experience in 
PACOM activities (Prof Mike Sovereign and Mark Sinclair) and Bosnia (Terry Prosser). 
Dr Russ Richards participated as a member of the Synthesis Working Group. 


Operations Other Than War: 
Background 


1996- PACOM Workshops 

January 1997 - MORS OOTW Workshop 

October 1997 - HA/DR IN THE NEXT CENTURY 
- CCRP report 
- VIC Concept description 


April 1998 briefing “Peacekeeping in Bosnia- New Frameworks for 
Analysis” 


Background: 


Previous workshops and activities provided groundwork for this working group. Several of the 
participants had been involved in the PACOM-sponsored workshops addressing OOTW analysis 
held in 1996. In January 1997, MORS also conducted a workshop on analysis methods and tools 
for OOTW. The C4ISR group characterized the complexities posed by the OOTW missions, and 
cited some of the differing analysis requirements and the need for tools to support analysis. 


Further background work was provided via a report entitled Humanitarian Assistance and 
Disaster Relief in the Next Century. (DoD C4ISR Cooperative Research Program, October 28-30 
1997). This report, written by Professor Mike Sovereign, documented working groups focused in 
three main areas: collaboration, cooperation, and coordination (C3) for HA/DR; information 
acquisition for HA/DR; and crisis management in HA/DR operations. Mark Sinclair gave an 
overview of the Virtual Information Center concept conceived in the workshop and demonstrated 
in PACOM. The VIC concept proved valuable to support crisis planning and allowed innovative 
analysis and problem solving in these missions. 


Finally, Terry Prosser presented a briefing entitled “Peacekeeping in Bosnia —- New Frameworks 


for Analysis” that reflected the issues and measures he was faced with while serving as analyst on 
the Bosnia task force in 1997. 


These inputs provided a focused starting point for the efforts of this working group. 


Operations Other Than War: 
Development Approach 


Development Approach: 


In order to address its assigned tasks, the group worked through a decomposition process 
depicted on this slide. For the set of OOTW missions, we first derived a set of typical 
functions within the missions and identified typical mission scenarios for analysis. For 
the scenarios, we considered how C4ISR means were employed in the mission. By 
crossing the C4ISR means with the mission functions, we were able to identify the most 
interesting C4ISR evaluation issues. Finally, these led to a categorization of typical 
measures of merit and the appropriate tools to assess them. The results of the process are 
presented in the following slides. 


C-5 


Operations Other Than War: 
Functional Characteristics of OOTW 


Peacekeeping 

Peace Enforcement 

(Peace Buitiding) 

Humanitarian Assistance/Disaster Relief 
Civil Support 

Nation Assistance 

Counter-drug 

Counterinsurgency (support) 


Mission/Functional Characteristic Matrix: 


The decomposition step identified the twelve major functions associated with OOTW 
missions. The missions considered were peace keeping, peace enforcement and peace 
building (PK/PE/PB), humanitarian assistance and disaster relief (HA/DR), civil support, 
nation assistance, counter drug and counterinsurgency. By agreement, any other mission 
in which combat might be anticipated was left to the Small Scale Contingency working 
group. 


From this decomposition, it is easy to assess where C4ISR means may be significant to 
mission accomplishment for each mission. Because of the limited time to the workshop 


and the particular expertise of the group, we focused on the PK/PE/PB and HA/DR 
mission areas. 
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Operations Other Than War: 
OOTW Mission Characteristics 


ὁ Characteristics of PK/PE © Characteristics of HA/DR 


Focus Different : Focus Different: 
® Prevent conflict © Stop the dying 


Restore governance 


& 

— Variety of participants 
® Stabilize Situation ere Ρ 
Φ 


(players) 
Exit — Operations last beyond US 
Military participation 
— Multiple participants — Small footprint 
— Operations last beyond US — Definition of winning is 
Military participation different (“hearts and 
— Security operations are minds’’) 


Primary 
Level of force varies 


‘Different ) Missions - . Different Issues: Similar Problems to Address _ 


OOTW Mission Characteristics: 


In order to get to the measures of merit necessary for analysis, two subgroups were formed. 
One group considered the Humanitarian Assistance/Disaster Relief mission area and the 
other considered Peacekeeping and Peace Enforcement. In each case, the subgroup listed 
some of the mission objectives and characteristics that would drive analysis objectives. 


In each case, it was clear that the main mission objectives were quite different than the 
mission objectives of mid-intensity warfare. In the case of PK/PE, mission objectives 
focus on security operations, prevention of conflict, and rebuilding of governance and 
economic stability. On the other hand, the focus of HA/DR is on emergency support, 
including objectives like "stop the dying.” 


In both cases, missions involve multiple participants, usually beyond the US military. In many 
cases, a small footprint in desirable, though expeditious completion of the mission is also desired. 
And the forces necessary for mission completion will be tailored mix. In the case of PK/PE, 
combat and combat support forces capable of preventing combat and providing security are 
important. In HA/DR, engineers, medical support are appropriate. 


These observations mean that the measures used in analyzing and assessing these missions will 
both differ greatly across the missions and will also differ greatly from the type of measures used 
in theater warfare. This will be demonstrated in greater detail in the next slides. As might 

be expected, analysis of C4ISR within these missions will pose a variety of paercnene 
substantially different than those faced in theater warfare. 
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Operations Other Than War: 
Example Decomposition - PE/PK 


CLASS MOE DATA SOURCES SYSTEMS 


Mission Freedom of Movement Define boundaries, # Observations & HUMINT 
Accom- crossings Cameras SIGINT 
plishment Civil Affairs 
Police H/R violations Define & Count Media ἃ Reports [ Public 
Compliance violations Define & Count leon 
Reports 
Polling results Conduct polls, collate PSYOPS & cn ies 
; : aneuver 
Public Affairs Milita 
Government Sovereignty Define, Count & Monitor 10 & Reports Polic a 
Media trends Collect, review, & collate Civil Affairs & Signal Corps 
Public Affairs INTEL 


Economic trends Market Analysis International ὑδραὐω 
Organizations & | ΒΕ 
Air Traffic 
Counters Control 
Health trends Disease Rates IO & indigenous : 
Medical 
reports 


Cost 


Force Threatcon Monitor Intel / Sensors / 
Protection Reports 


Casualty trends Monitor / Report Forces / 
Reachback 
Monitor / Report 
Credibility Monitor / Report Forces / Intel 


Example Decomposition -- PK/PE: 


These two slides show decomposition for the PK/PE mission area. This particular 
decomposition is based on experience with the Bosnian situation, but it is worth noting that the 
particular result would change for different situation. 


General areas of interest are in mission completion measures, force protection measures, 


decision making measures, and interoperability measures. Within these areas, the 
decomposition addresses measures that are appropriate for C4ISR tradeoffs. 


C-8 


Operations Other Than War: 
Example Decomposition (cont’d) - PE/PK 


CLASS MOE DATA SOURCES 
Decision Making | Difference between perception ἃ | Uncertainty / Risk Historical 
truth Mitigation Assessment 
Right info ? Trend & Pattern 
analysis 


Timely info ? Reaction time} 
< Planning 
Decision time} 
Horizon 


Interoperability # Interpreters required 
Network Connectivity Systems, Networks 


Communications Architecture 
Complexi 
orders 


COP 


Responsiveness Response Time 


Example Decomposition -- PK/PE (cont’d): 


The mission completion category includes measures that address stability in the area of 
operations at the “measure of force effectiveness” level. The sort of measures developed (e.g. 
freedom of movement, economic and health trends) are diverse and may involve C4ISR in their 
completion. Therefore, C4ISR trades are viable and indeed important. Notably, they are 
challenging to analyze because tools to address these measures are scant. In the theater, current 
experience is to assess progress by trend analysis of data actually gathered. In planning cases, 
tradeoff analyses could be conducted by comparing potential effectiveness and cost of using 
alternative collection means. Trades based on impact on economic or cultural response in 
theater would currently be nearly impossible to assess. 


Force protection measures lend themselves to tradeoff analysis since they can be addressed as 
effectiveness and cost issues. On the other hand, trades are more difficult to assess in the 
context of decision making support, since the scenarios and models are not widely available. 
Perception-based models are lacking in OOTW, just as they continue to be lacking for war 
fighting missions. In the area of interoperability, several measures can be addressed 
analytically since they are more closely related to network performance. 


The sort of decomposition presented here illustrates some of the needs and possibilities for 
C4ISR tradeoff analysis. This example will apply for many situations, but is certainly not 
complete. Further work to extend these results is warranted. 
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Operations Other Than War: 
Measures of Merit 


© HA/DR | 
Internal (among C4ISR systems) — Internal (among C4ISR systems) 
Φ Information Timeliness δ΄ Timeliness 


® — Info Completeness/ * —Completeness/Accuracy 
Accuracy * Footprint 
Network Completeness δ΄ Risk/Flexibility 
Footprint — External (between C4ISR and 
others, or between C4 and ISR) 


5. Mission Impact / $$ 
Architecture Complexity/ 6 
Interoperability 


Risk/Flexibility 


Efficiency of Operation 
5. — Perception of Operation 


Common Operational ὰ : 
Picture Reduction of 


Mortality/Morbidity 


External (between C4ISR and others, Non-surprise/Prevention 


or between C4 and ISR) 
® Mission Impact / $$ 
Efficiency of Operation 
Perception of Operation 
Political Economic Impacts 
Conflict incidents 


Measures of Merit: 


An overview of measures for PK/PE and HA/DR as applied to tradeoffs within C4ISR, 
and between C4ISR and other forces and systems is presented here. For trades within 
C4ISR, the measures frequently can be tied to measures of network performance. When 
this is true, typical analytic capabilities can be brought to bear. 


When trades are to be conducted outside C4ISR, the measures are more MOFE-level. 
They will tend to differ for each of the mission areas in OOTW. they also are not 
supported by well-developed tools and models to conduct the analysis. Mission 
efficiency, in particular, will be related to economic and cultural measures. The theory 
and “transfer functions” from C4ISR effectiveness (or any other system or force 
capability) are lacking in most areas. 
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Operations Other Than War: 
Tools 


Analytic Purpose Analytic Tools 


Situation Assessment 


Polling 
© Identify trends and patterns © Statistical Analysis 
Correlate across classes of MOEs Φ ; 
Sampling 
Political/economic/mili 
Sane aa © Hypothesis Testing 
Ethnic/cultural/religious © Correlati 
orrelation 


Infer cause and effect e : 
Data display 

Track progress towards objectives e i 
Data from past OOTWs/data mining tools 


© OOTW IPB 


Support to Decision Making and C2 


" Knowledge-based expert systems 
© Analyze alternative courses of action 
© Panels of experts 
© Identify requirements 
® — Virtual Information Centers 
Reduce uncertainty | 
τον ἦς © Wargaming 
Improve timeliness 
. ive decisi king tools 
Assess quality of interoperability and comms Collaborative decison making too 
© Network analysis tools 
© Functional capability models 
© Lessons learned databases 
Trade-Off Analysis e : 
Analytic frameworks 
9 Support QDR-like force trades ο s ? 
cenarios 


© Understand Total Force implications of OOTW 


© Data from past OOTWs 
Expert judgment 

Predictive capability models 
© OPTEMPO models 


Tools: 


This slide portrays the types of tools that are needed to analyze OOTWS. It is organized in 
terms of major analytic purposes. The first two purposes (situation assessment and support 
to decision making and C2) are inherent in the planning and conduct of OOTWs. Trade- 
off analysis addresses the broader issue that the workshop was charged to address: how to 
trade-off C4ISR against other categories of things, such as forces, to support QDR-level 
force trades. 
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Tools (cont’d): 


While the tools that support these different purposes are organized into separate "bins" on 
the slide, in fact there are broad analytic needs that cross these categories. These include: 


° Better ways to gather, analyze, and display data 
¢ An understanding of past OOTWs and lessons learned 


5. Ways of tapping expert judgment. A promising example is the Virtual Information 
Center (VIC) concept for HA/DR being developed for PACOM 


5. Analytic frameworks for OOTW 


While an end-to-end modeling approach to OOTW is well beyond the present state of the 
art, there is a clear need for tools and models that will do part of the job within the context 
of an analytic framework. For example, tools for HA/DRs that help to predict damage and 
casualties and generate requirements for medical supplies and personnel are invaluable. 
Within peace operations, models for assessing the performance of difference ISR 
capabilities or communications networks would be similarly helpful. 
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Operations Other Than War: 
Relative Worth/Trade Offs 


¢ 6Trade-off taxonomy 


— Internal - within C3 and ISR 
— Internal - between C3 and ISR 
— External - between C4ISR and everything else (e.g., forces) 


¢ Trade-offs are progressively more difficult from internal to external 


— MoMs go from relatively technical and quantifiable to 
operational/political and qualitative 

— Cause and effect relationships supporting higher-level MoMs become 

problematic 


Relative worth/Trade-offs: 


Trade-offs within the categories of ISR and C3 can be done in traditional ways that apply 
to other military operations. Measures of performance and effectiveness in these areas 
(such as quality, quantity, and timeliness for ISR) are well understood and relatively 
quantifiable. When one tries to trade-off ISR for C3 or, more broadly, C4ISR for other 
things such as maneuver forces, the capabilities are unlike, and causal relationships leading 
to the achievement of measures of force effectiveness or measures of policy effectiveness 
are highly uncertain. For this reason, the trade-offs that commanders make during every 
OOTW tend to be qualitative and informed by expert judgment rather than quantitative 
analysis. Qualitative trade-off judgments can be just as good as quantitative ones. In fact, 
in this area, they may be better than judgments based on trying to force the numbers to fit 
the problem. However, the ability to make qualitative trades, whether for operational or 
force planning, can be significantly improved by enhanced capabilities in the areas 
indicated by the slide: 


e¢ Definition of missions and requirements 

e Exploitation of past lessons learned 

© Databases of operational data 

9 Frameworks for analysis with tools and models integrated as appropriate 
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Operations Other Than War: 
Relative Worth/Trade Offs (cont’d) 


° Kasier trades (e.g., within ISR) may be helpful for programmatic and mission 


planning issues but are less likely to shed light on QDR-level force trades 
° Trades of C4ISR with forces for OOTW require: 


— Better definition of missions and requirements 
— More organized exploitation of past lessons learned 
— Databases of operational data 
Frameworks for analysis and appropriate supporting tools 
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Operations Other Than War: 
Assessment of Current State-of-the-Practice 


Assessment of Current State of the Practice: 


It is not surprising to see that the state of the practice assessment shows many areas of 
shortfall. 


Structuring of the Problem rated a Yellow/Red. Recalling the focus of conducting 
analysis and tradeoffs of C4ISR assets, we see that there are few developed tools and 
procedures to apply in the general area of OOTW. Nevertheless, we acknowledge 

that these missions are being continually addressed day-to-day with apparent success. 
Human factors and organization rate a red because the human elements of C4ISR are 
not well modeled in OOTW. The human performance aspects of C4ISR performance 
metrics are less difficult than the responses of the actors in the OOTW area of 
operations. That is, the connectivity from C4ISR actions to cultural, religious, or other 
reactions is lacking, limiting the ability to conduct MOE-based analysis. 

Scenarios rated a yellow/green. The reason is simply that recent history has provided 

a large number of plausible planning scenarios. During the last QDR, a body of 
approximately 50 scenarios was developed for the Dynamic Commitment wargame 
series. The yellow results because the scenarios are not well documented and available 
for planning use at this time. 
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Assessment of Current State of the Practice (cont’d): 


Measures of Merit is rated as red, tending toward yellow. The workshop efforts as well 

as the operational missions over the last few years have now organized this area of 
investigation. Progress is being made in identifying these measures. Although the 
diversity of the OOTW missions makes this more difficult, trends are starting to emerge 
regarding what types are appropriate. 

Tools and applications rate red, trending toward yellow. In general, these missions are 
diverse and complex enough that no monolithic model is viewed as feasible or desirable. 
Rather, what is desired are sets of tools that address parts of the analyses directly and can 
be intelligently composed to address more complex questions. A few mission and force 
building tools are available and some C4ISR issues can be undertaken with standard 
network tools. However, the connection from C4ISR performance to mission effectiveness 
is missing, as was indicated under structuring the problem. 

Data is considered red. Evidence is found in the difficulty to access any data associated 
with any of the recent OOTW missions the US has been involved with. Furthermore, 
operational planning continues to be hampered by difficulty in rapidly accessing useful 
data. Due to the difficulties with measures, tools and data, it follows that areas like risk 
assessment would also rate red. Indeed, finding the marginal benefit or the risk associated 
with any analysis topic tends to be a difficult task. 

Finally, reporting rates a red trending toward yellow. Until recently, OOTW reports were 
hard to come by. During the QDR, efforts to assess past efforts were hampered by scant, 
inconsistent data and reports. Recent efforts have included greater efforts to gather data 
and report results. Further work is required. Mission objectives, forces, timelines, MoMs, 
as well as other data and analytic needs should be articulated so that future reports can 
provide useful information for the future. 
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Operations Other Than War: 


Summary Observations 

e Diverse OOTW add to complexity 

— Of the mission area 

— Of the design of analysis tools 

— Of C4ISR tradeoff development 
e Mission MoMs: 

— Are known, but vary widely by mission 

— Are “standard” for many trades within C4ISR 

— Are very difficult for tradeoff with Forces 


° Not “ground lost” or “casualty ratio” 
e Tradeoff Analysis focus varies . 


— OQOperational-level (JTF) analysis most common 
— PPBS/Acquisition-level lacking 


Historical databases lacking 


Summary Observations: 


The working group made several general observations in the course of its efforts. Many of them 
support the preceding discussion, but others need to be brought out separately. 


First, we reiterate the fact that the diversity of OOTW missions adds complexity to C4ISR analysis. 
The breadth of the missions, and their different nature adds to the complexity in developing general 
approaches to conducting C4ISR analysis and complicates the process of designing tools to aid 
analysis. Then, we observe that the operational experience base exists to allow identification or 
development of MoM. These MoMs are known, but vary widely by mission and are definitely not 
“ground lost” or “casualty ratio.” Many C4ISR-related trades lead to MoMs that are “standard” 
for such analysis. However, we see that MOFE measures are more difficult to assess in tradeoffs 
with other systems and forces. 


We also note that we found it natural to discuss mission and tradeoff analysis in the context of 
JTF-level operations. Most of our discussions focused at that level. We see that these trades could 
easily be extended to acquisition-related system trades. However, for PPBS or QDR-level analyses, 
we think additional methodology is necessary since no single OOTW mission is likely to be the basis 
for force structuring or capability analyses. 


Finally, our collective experience was that data access and historical data to support operational 
planning and analysis has been limited and limiting. 
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Operations Other Than War: 
Recommendations 


Establish an effort to advance our knowledge of OOTW and C4ISR in OOTW 


— Leading to analytic frameworks 
— Best Practices 


Continue efforts to ID analytic tool requirements to assess C4ISR support to 
OOTW 


Continue development of methods to better integrate OOTW analysis into 
DoD planning & programming | 


Establish a common syntax & semantics to facilitate a comparison between 
study results 


Recommendations: 


The working group was able to address the issues specified by the Terms of Reference for this 
meeting. The outline for generating MoMs and structuring C4ISR-centered analyses provides a 
useful step, but needs to be extended in both breadth and depth. As a result, we make the 
followmg recommendations: 


Continue this process of methodology development. It should be a general effort to advance our 
knowledge of OOTW and C4ISR in OOTW,, leading to analytic frameworks 

and best practices. This effort should also continue the process of identifying analytic tool to 
assess C4ISR support to OOTW. 


At the DoD level, this effort should continue development of methods to better integrate OOTW 
analysis into DoD planning & programming. This need largely is derived from the fact that any 


single OOTW is too small and narrowly focused to drive force structure or major programming 
decisions. 


Another aspect of these developmental efforts is to establish a common syntax & semantics to 
facilitate a comparison between study results. Such a framework will facilitate reviewing, 
comparing various missions and analyses, and will encourage more broadly based analysis. 


Finally, a data support process is required to enable routine analysis. Procedures should be put in 
place to ensure data is gathered, stored and made available for analysis efforts. 
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Operations Other Than War: 
Recommendations (cont'd) 


¢ Develop a deliberate approach to data collection 


ID & prioritize needs 
— Establish collection mechanism 
— Establish repositories 

Include Non-DoD sources 
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Infrastructure Assurance: 
The Problem 


How does C4ISR support Infrastructure Assurance? 
— What is C4ISR? 
— What is infrastructure assurance? 
— What are the infrastructures? 
What are the infrastructure assurance information requirements? 


How can we analyze the C4ISR infrastructure to improve support of 
infrastructure assurance mission area? 


Working Group Tasking: 
Here is the problem that the working group tried to address, and a partial answer: 


In particular, how does C4ISR support Infrastructure Assurance? 
e What is C4ISR and how does it fit into the context of Infrastructure 
Assurance? 
e What are the physical and information requirements? 
e And how can we analyze and use the C4ISR infrastructure to improve support 
of the Infrastructure Assurance mission area? 


What is Infrastructure Assurance? 
Infrastructure Assurance is the planning to improve the readiness, reliability and 
continuity of infrastructures such that they are: 
e Less vulnerable to disruptions or attack: 
e Harmed to a lesser degree in event of disruption or attack; and 
e Can be readily reconstituted to re-establish vital capabilities. 


Infrastructure Assurance: 
Characterize C4ISR 


e Infrastructure Assurance/Critical Infrastructure Protection (CIP) is new and 
evolving mission area 
— PDD 63 signed May 1998 
— Our C4ISR knowledge from conventional warfare does not 
necessarily apply to infrastructure warfare 
Current DoD CIP Infrastructure Sectors 


Defense Info. Infrastructure ¢ —_- Transportation 
Φ Command, Control and © = _ Health Affairs 
Communications ¢ Emergency Services 
9 Intelligence, Surveillance and ® Personnel 
Reconnaissance ° Space 
Public Works ° 


Logistics 
Financial Services 


C4ISR must broaden to include law enforcement, OGA and the private 
sector 


Characterize C4ISR: 


Infrastructure Assurance, until recently referred to as Critical Infrastructure Protection 

(CIP), is a relatively new and evolving mission area: 

e Presidential Decision Directive (PDD) 63, which presents the overall scope of Critical 
Infrastructure Protection and calls for a national plan and structure to cope with the 
problem, was just signed in May of this year. 

e It addresses a new kind of warfare, unlike what we have experienced in conventional 
warfare. 

e To work the problem, the draft DoD CIP Plan divides DoD mission areas into 11 
infrastructure sectors 

e 3 of those sectors comprise the C4ISR mission area 

e Each of the 3 C4ISR sectors must address all the physical, as well as cyber- 
related assets within that sector. For example we are not just looking at 
platforms and systems. 

e Additionally, these sectors cannot be considered in isolation from one another. 
Each is often dependent on the other to achieve its mission objectives. (e.g., 
electric power to run the computers, water to cool the computers, etc.) These 
services are routinely provided by commercial assets outside the government 
sector. 


Therefore, the C4ISR community must broaden its vision to their dependencies on other 
government agencies and the private sector. 
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Assets, Infrastructures, and 
Interdependencies 


pperations 
; Plan ὁ 


vate Infrastructure, ἐς. 


Assets, Infrastructures, and Interdependencies: 


As we tried to demonstrate here, there are a myriad of interdependencies between the 
critical assets within certain Defense Infrastructures, other Defense Infrastructures, and 
the Private/Commercial Sector Infrastructures. 


These dependencies and value chains must be well understood for analysis within the 
DoD lifecycle approach to the CIP problem (as defined in the draft DoD CIP Plan). 
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Infrastructure Assurance: 
The DoD Strategy 


Approach: 


--ὀ Identify Relevant Characteristics of Each Infrastructure that is 
Critical to Military Mission Success 


— Analyze Military Plans to Identify Critical Infrastructure Assets 
~- Assess Vulnerabilities of Critical DoD Assets 
— Redirect Resources to Reduce Prioritized Vulnerabilities 


— Collaborate with Other Government Agencies and the Private Sector 
to Reduce Vulnerability of Private Sector Infrastructure Assets 
Critical to DoD Mission Success 


The DoD Strategy: 


The DoD approach to the CIP or Infrastructure Assurance problem is to: 


e Identify relevant characteristics of each infrastructure that is critical to military 
mission success. 

e Analyze military plans to identify critical infrastructure assets. 

e Assess vulnerabilities of critical DoD assets. 
Redirect resources to reduce prioritized vulnerabilities. 

e Collaborate with other government agencies and the private sector to reduce 
vulnerabilities when services are provided from outside the DoD fence. 


This should be done in the context of six (6) lifecycle phases that occur before, during 
and after events which may compromise or disrupt the services or mission 
accomplishment of a critical infrastructure. 


Analysis, Assessment and Remediation are proactive preventive measures that take place 
continually. Indications and Warning primarily occur before an event. Mitigation occurs 
both before and during an event. Response occurs during an event and Reconstitution 
may start during an event, but is generally concentrated afterward. 
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Infrastructure Assurance: 
C4ISR Must be Able to Support the IA Life Cycle 


Example: Analysis/Assessment 


Current Requirements 
— Architecture 

° Example: JCAPS, ISAAC 
— Data 


5. Commercial Infrastructure (Electric Power, Communications 
Transportation) 


? 


— Example: NCS for Communications 
5 Cyber Infrastructure: NIPC Certs 
Future Requirements 
— Data 


° Industry-Government Information Exchange 
— Wargaming/Simulation 


C4ISR Must be Able to Support the IA Life Cycle: 


This and the next graphic are examples of how C4ISR possibly can support current and 
future infrastructure assurance requirements within a given lifecycle phase. 


For the Analysis and Assessment Phase, operational architecture data on organizational 
missions, functions, structures, systems and their relationships to include information 
flows and dependencies can possibly be provided by such tools as the Joint C4ISR 
Architecture Planing/Analysis System (JCAPS) and the NCS for communications. For 
Cyber Infrastructure, the NIPC and various Certs are C4ISR components that support 
analysis and assessments. Future requirements will necessitate an exchange of industry 
and government information for wargaming and simulation exercises. 


Infrastructure Assurance: 
C4ISR Must be Able to Support the IA Life Cycle 


Example: Indications & Warning 


Current Requirements 
— Inter-Agency Information Sharing 
¢ Example: MDITS (Counter-terrorism) 


— Indicators (Criteria) for Minimum Operating Capability for Cyber 
Attack only 


95 _ Future Requirements 
— Government—Commercial Monitoring ἃ Warning 


— Indicators (Criteria) to Assess Status of Affected Infrastructures, 
Operations, or Assets 


Trace Ability 


C4ISR Must be Able to Support the IA Life Cycle (cont'd): 


For the Indications and Warning Phase, inter-agency information sharing processes will 
need to be developed to satisfy current requirements. MDITS is an example. 


Future requirements will necessitate: 
e Government — Commercial Monitoring and Warning 


e Indicators to assure status of affected infrastructures, operations, or assets, and 
e Trace ability to the root of the problem. 


Infrastructure Assurance: 
Tools: Current Capability 


DoD C4ISR tools generally lack the capability to facilitate detailed IA 
analysis because C4ISR requirements generally have not addressed the need 
to support Infrastructure Assurance. 


IA tools have been developed in response to Infrastructure Assurance 
community requirements and are effective for analyzing commercial 
infrastructure (power, transportation, communications, and information 
systems) and the military missions they support. 


Real-time infrastructure analysis hindered by poor integration of C4ISR 
and IA analysis tools 


Tools: Current Capability 


Current tools capabilities and additional tool requirements are presented on this and the 
next graphic. 


DoD C4ISR tools are currently lacking the capability to facilitate detailed infrastructure 
analysis. Therefore, separate tools have been developed to meet individual or specific 
needs. These tools are effective in the Infrastructure Assurance community for analyzing 
power, transportation, and communications. However, real-time infrastructure analysis is 
hindered by little or no integration of C4ISR and Infrastructure Analysis tools. 
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Infrastructure Assurance: 
Tool Requirements 


e IA Tool Needs: 


— Integrated and Interoperable Tools 


— Comprehensive Models and Tools for Analyzing Defense 
Infrastructures 


— Models and Tools for Interdependency Analysis 


e Need Collaborative Link between Infrastructure Analysis Centers and 
Warfare Centers 


Tool Requirements: 


What is needed are integrated and interoperable tools, as well as comprehensive models 
for analyzing Defense Infrastructures. Models and tools are also needed to conduct 
interdependency analysis. 


We also need to establish collaborative links between infrastructure analysis centers and 
warfare centers. 
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Infrastructure Assurance: 
Relative Worth 


Possible C4ISR Contributions: 


5 Analysis and Assessment Tools 
¢ Near Real Time Reporting Across All Sectors 
— T&W 


— Infrastructure Status 
Alternative Courses of Action 
Synchronized Information that Identifies the Criticality Attribute (e.g., 
Time) of Assets 

Information Dissemination and Sharing in Support of Infrastructure 
Assurance 


Relative Worth: 


As suggested above, we believe the C4ISR community can provide some of the tools for 
architecture development and information sharing in support of Infrastructure Assurance. 
It also may provide some of the tools needed for near-real-time reporting across all 
sectors, particularly with respect to I&W and infrastructure status. Other C4ISR tools are 
available to provide information on alternative courses of action and synchronized 
information that identifies the critical attributes (e.g., time) of assets. 
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Infrastructure Assurance: 
Measures of Merit 


Measures of Social Continuity/Health (e.g., 


Qualitative: Population Confidence; 
Quantitative: Delta in Dow Industrial Avg 


Measure of Mission Effectiveness (e.g., Success 
DoD’s Ability to Mobilize) 


Measures of Functional Performance 
vl | (e.g., Attack Assessment; Accuracy, 
“...¢2; Completeness) 
Measure of System Performance 
(e.g., Timeliness, Accuracy, 
Covertness) 


το ἢ GAISR Systems ἢ SN 


Measures of Merit: 


We suggest four hierarchical measures of merit with respect to Infrastructure Assurance 
and its dependence on C4ISR as a basis for mission success and national security. At 
each level, the measures of merit must be consistent. Thus, for example, Infrastructure 
Assurance functions (“the lifecycle” activities of Assessment, Remediation, I&W, 3 
Mitigation, Response and Reconstitution) depend on our ability to perform timely, : 
complete, accurate detection and assessment of an attack. To do this, the C4ISR systems 
upon which Infrastructure Assurance depends must meet system performance measures, 
such as 

e timely; 

e accurate; 

e global collection of threat; and 

e event data; 
to support attack assessment and reporting back to affected locations, without raising 
suspicions of the attackers. 
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Infrastructure Assurance: 
Key Issues and Questions 


Definitions & Distinctions 


What is a Critical Asset? 
Infrastructure Protection vis-a-vis Force Protection 
Reliance On Host Nation Infrastructures 


Buy-in 

Data - “Achilles Heel’’ 
Security 
Legal and International 


Key Issues and Questions: 


Issues within IA revolve around internal discussions and decisions such as definitions and 
distinctions. For example, what really is IA? Where do we draw the line between 
infrastructure protection and force protection? 


Issues that are internal and external revolve around "Buy-In," that is, getting our 
governmental entities both within DoD and other Federal, state and local agencies 

e to share information 

e to have acommon understanding of IA 


Buy-In also includes government to commercial industry issues: 
e sharing proprietary information 
e who pays? 


Issues of immediate relevance to MORS and the C4ISR question revolve around data, 
such as: 
e Fidelity 
¢ Getting Commercial Sector to Provide Data (Often Proprietary) 
e Getting Government Organizations to Provide Data (Asking vs. 
Tasking) 
e Central Data Repository 
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And of course, there are Security issues too broad and too deep to discuss at this point, 
but obvious from the C4ISR perspective and now the JA perspective. 


There is a substantial CONUS-focused legal question. And there are substantial 


international issues, again too broad and too deep to discuss at this point, but relevant to 
the future of IA. 
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Infrastructure Assurance: 
Recommendations 


MORS continue to emphasize 
Infrastructure Assurance as an 
essential element of C4ISR (and 
other) analyses and wargaming 
and simulation exercises 


Joint C4ISR and Infrastructure 
Assurance community analysis of 
the application and integration of 
complementary tools 


Recommendations: 


While there are quite a few recommendations one can see deriving from the C4ISR-IA 
connection, we have focused on two (2). 


e The first is that MORS and its connections to the education community, such as 
the National Defense University, continue to actively pursue the inclusion of 
Infrastructure Assurance in its modeling/simulation assessment activities. 


e And the next is that we in the IA community and the C4ISR community begin 
to develop mutual understanding of our requirements in order to develop 
integrated tools systems, etc. 
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Infrastructure Assurance: 


COBP Analysis 


ἊΣ 


‘emplate 


Problem 
Structuring 
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Peacetime Engagement (PE): 
Working Group Members 


Bruce Powers (Chair) 
Dean Free (Co-Chair) 
Kevin Brandt 

Trena Covington 


LCDR Marty Keutel, 

Dr Steve Pilnick 

Dr Dave Schrady 

CDR Mark Tempestilli, USN 
Chris Vogt 


Office of CNO, N816 
Office of CNO, N816 
MITRE Corporation 


Johns Hopkins University, 
Applied Physics Lab 


USN Office of CNO, N6C 
EDO-Technology, Inc. 
Naval Postgraduate School 
Office of CNO, N6C 


Navy Warfare Development 
Command 


Peacetime Engagement (PE): 
Presentations 


“Maritime Battle Experiments,” Chris Vogt, NWDC 
‘Network Centric Warfare Concepts,’’ Dean Free, N816 
“Navy RMA 2020 Wargame Results,” CDR Bill Valentine, N81 
“Evaluating Mission Effectiveness in OOTW,” Adam Siegel, CNA 


“Fconomic Benefit of Forward Presence,’ Dr David Schrady, NPS 


“Design Considerations/MOE for Linking C4ISR with M&S,” 
Kevin Brandt, MITRE 7 


“Net Collective C’,’? CDR Randy Bowdish, Office of CNO, N51 


Peacetime Engagement (PE): 
Definition 


The application of resources (including forces) in peacetime to make shooting less 
likely. Successful PE would promote other goals: 


Stability of commerce 

Deterrence 

Separation of potential belligerents 
Exercises with (potential) allies 


Definition: 


5. Nocurrent Joint Staff definition of Peacetime Engagement (PE). 
° Current related Joint Staff definitions: 


— Overseas Presence: The totality of US military instruments of power deployed 
overseas. It is that portion of the force postured overseas to support performing 
the full range of military activities comprised of permanent forces, temporary 
forces and infrastructure. 

— Engagement: All military activities involving other nations intended to shape the 
regional security environment in peacetime. Engagement is comprised of 
operations, exercises and other foreign military interactions. 


The above definition characterizes PE as deterrent in nature and includes benefits of 
deterrence as broad goals that are measurable to some degree. 
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Peacetime Engagement (PE): 
Pre-Conflict C4ISR Performance Measures 


¢ Threat located 


% of total OOB 

% of number deployed OOA 
% in vicinity of blue opareas 
% threat weapons close 
False target rate impact 


Φ 9% needed info available 


— At partners’ command centers 
— At US command centers 


Pre-Conflict C4ISR Performance Measures: 


These represent a sample of C4ISR measures in the pre-conflict phase of a campaign. 
One would also need to consider: 


e Threat intentions 
e Threat strategy 
4Φ Centers of gravity for the threat 


9 Courses of action to deter the threat —- need to understand our options and the 
threat’s response; “effects-based” options, not just an attrition-based approach 


Bottom line: ISR is MUCH more than a collection of numbers. 


Peacetime Engagement (PE): 
Lessons Learned 


C41 is a pillar of PE 
Needs to be adaptive to PE circumstances 


Installing, operating, commanding and maintaining network may become 
the most important functions 


Less easy to predict C4I patterns 15 years hence than it is to predict 
platforms in inventory 


Info ops highest utility will be in PE 


Lessons Learned: 
Discussion led to the list of lessons learned on this and the next two slides. The 


discussion centered around the insights provided by the seven briefings received by the 
working group, plus discussions of the purpose and objectives of the workshop. 
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Peacetime Engagement (PE): 
Lessons Learned (cont’d) 


Intel prep of battlefield is conducted during PE 

PE provides means to partially validate models of warfare 

PE not a lesser included case of warfare 

Greater use and size of C4I systems may bring greater C4I vulnerability 


Deploying C4ISR increasingly follows model of naval deployment 


Peacetime Engagement (PE): 
Coalitions 
Greater number of potential partners in PE than in specific MTWs 


Interoperable C4I required for coalitions 


Reliance on coalition wartime capabilities requires peacetime investment in 
C4] 


Coalition interoperability improves war fighting capability 


Coalitions: 


These are conclusions about coalitions that resulted from discussion of PE in the broader 
context. 


Peacetime Engagement (PE): 
Key Issues and Questions 


1. After Cold War, the spectrum of potential conflict has broadened 
— The probability of occurrence of events on this spectrum may be 


differently distributed 
— Data collection and analysis of that issue should follow 


2. Naval Postgraduate School research suggests major benefits ($10Bs) in 
savings on oil prices by prompt deployment of naval forces in crises. Can 
the scale of benefits for deployment of C4ISR be similarly identified (e.g., 
AWACS deployment impacting prices of drugs from Colombia)? 


Key Issues and Questions: 


1. The nature of the world, the threat, has changed drastically since the end of the cold 
war. The distribution of risk of warfare is quite different, with more uncertainty 
about the probabilities of specific threats. There is a need to collect data and develop 
the distribution of events on the current and future spectra of conflict. 


2. The Naval Postgraduate School study identified significant potential savings on oil 
prices by early deployment of naval forces. It is possible that early deployment of 
C4ISR systems could provide similar benefits. 


Peacetime Engagement (PE): 
Key Issues and Questions (cont’d) 


3. The less an event looks like war, the less able we are to articulate its 
governing processes. C4ISR in PE requires drawing in political 
constraints/considerations to a degree greater than for other points on the 
spectrum. This suggests peculiar data requirements for military/C4ISR 
commanders in PE settings. 


- For PE events, the cyclical process (define MOM, measure sensitivity, 
collect data, evaluate data, report) needs to also capture critical C4ISR data 
and effects. 
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Peacetime Engagement (PE): 
Recommendations 


Develop broadly-accepted TOR for PE: definitions, objectives, goals, 
distinguishing relationships from 
OOTW to MTW 


Develop CSOP for PE 
Develop MoMs and data collection checklists for PE 


Expand JULLS formats and databases to include C4ISR data 


Recommendations: 


Φ Recognizing PE as an important mission area, with attendant principles and 
distinguishing relationships, will assist in identifying the peculiar tasks of PE, and 
garnering funding for their successful undertaking. 


¢ Common Standard Operating Procedures must account for the many differences 
(historical, cultural, financial, educational, etc.) that exist in the many countries that 
we engage during PE operations. “One size fits all” is not the answer. We need to 
KNOW our PE “targets” much better and tailor each engagement to that particular 
country. Specific C4ISR exchanges are good candidate elements for tailored PE. 


¢ Expansion of the Joint Uniform Lessons Learned database to include C4ISR data and 
information will help long term analysis of C4ISR issues. 
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Peacetime Engagement (PE): 
Recommendations (cont'd) 


° Study the impact of deploying C4ISR in peacetime 
— Assess utility of C4ISR deployment on its own 


— Assess how early C4ISR deployment facilitates early combat power 
by follow-on forces 


Recommendations (cont’d): 
The results of such a study, successfully completed, should help determine the value 


added by C4ISR systems. The study should also address how a robust C4ISR exchange 
program helps build coalitions and therefore serves as a significant deterrent. 
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Peacetime Engagement (PE): 
Assessment of Current State-of-the-Practice 


Human Factors & 
ες Organization. - 


Assessment of Current State-of-the-Practice: 


Scenarios represent the only good news on this slide. In all other areas, assessment of the 
current state-of-the-practice is discouraging. There is much work to be done to develop 
measures of effectiveness and tools and to collect data relevant to proper analysis of 
C4ISR issues related to PE. 


Peacetime Engagement (PE): 


Summary 
KEY ISSUES 


° Since the cold war, the spectrum of potential conflict has broadened. The 
probability of occurrence of events on this spectrum may be differently 
distributed. Data collection and analysis of that issue should follow. 


The less an event looks like war, the less able we are to articulate its governing 
processes. C4ISR in PE requires drawing in political constraints/ considerations 
to a degree greater than for other points on the spectrum. This suggests peculiar 
data requirements for military/ C4ISR commanders in PE settings. 


For PE events, the cyclical process (define MOM, measure sensitivity, collect data, 
evaluate data, report) needs to also capture critical C4ISR data and effects. 


RECOMMENDATIONS 


Develop broadly-accepted TOR for PE: definitions, objectives, goals, 
distinguishing relationships from OOTW to MTW 


° Develop CSOP for PE 
5 Develop MoMs and data collection checklists for PE 
Expand JULLS formats and databases to include C4ISR data 


Summary: 


e KEY ISSUES 

e The nature of the world and the threat, has changed since the end of the Cold 
War. The distribution of risk of warfare is quite different, with more uncertainty 
about the probabilities of specific threats. There is a need to collect data and develop 
the distribution of events on the current and future spectra of conflict. 

. RECOMMENDATIONS 

e Recognizing PE as an important mission area, with attendant principles and 
distinguishing relationships, will assist in identifying the peculiar tasks of PE and 
garnering funding for their successful undertaking. 

e Common Standard Operating Procedures must account for the many differences 
(historical, cultural, financial, educational, etc.) that exist in the many countries that 
we engage during PE operations. “One size fits all” is not the answer. We need to 
KNOW our PE “targets” much better and tailor each engagement to that particular 
country. Specific C4ISR exchanges are good candidate elements for tailored PE. 

e Expansion of the Joint Uniform Lessons Learned (JULLS) database to include C4ISR 
data and information will help long term analysis of C4ISR issues. 
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Architectures 


Architectures Working Group 


Overview 


* Traditional analytical paradigms are platform-centric 
° Architectures allow you to treat C4ISR as a platform 
¢ Therefore allow evolution to ARCHITECTURE-CENTRIC analysis 


Overview: 


Analytical paradigms have traditionally focused on evaluating the effectiveness of a 
platform or system. Architectures provide a structured approach for describing and 
understanding a system-of-systems associated with a specified domain, such as C4ISR 
and in doing so allow that domain to be considered as if it were an entity. Using the 
architecture construct enables our analytical focus to evolve from being platform- 
centric to become architecture-centric. 


Architectures: Current Status 


Architectures are an evolving discipline 
-- part science and part art 

PROs 

e Provide an understanding of how information is used to accomplish work 
and how Information Technology supports work 

¢ ¥orm the basic data and data relationships that can support C4ISR issue 
analysis through various analytical techniques to include Modeling and 
Simulation 

CONs | 

ὁ Generally are static snapshots in time and hand crafted; difficult to reuse 


Current Status: 


While major architectural precepts are well defined and generally well accepted, architectures 
are a still evolving discipline which demand a combination of both art and science. 


From the positive perspective, architectures help us understand the relationship and 
dependencies among tasks, operational elements accomplishing those tasks and the information 
flow that needs to occur among the operational elements to accomplish the tasks appropriately. 
They enable us to describe the systems and communications that support that information flow 
across nodes and in doing so, enable the understanding of how Information Technology 
supports work. 


Architecture descriptions form the basic data set of activities, operational elements, 
information flows, nodes, systems and communications that support a wide variety of C4ISR 
issue analyses using various techniques to include Modeling and Simulation. 


The use of architectures may be limited or restricted because they generally provide a static 
perspective and represent snapshots in time. They traditionally have to be developed on an 
individual basis and the information is difficult to capture for reuse in related efforts. 


Architectures have too frequently been seen as an end in themselves when their true 
value is as a means to an end. A successful architecture effort must begin with a well- 
defined purpose and objective. 
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Architectures: Potential 


Architectures can provide a seamless bridge between the C4ISR 
user and developer communities. 


Operational architecture can capture operational requirements 
in a fashion that allows consistent traceability to design 
specifications. 


Symbiotic relationship between analysis and architectures 
which need to be emphasized and strengthened 


Potential: 


The traceable linkage across the architecture products provide an audit trail relating 
and translating mission effectiveness requirements of the C4ISR user to system 
requirements to which the system developer must build. 


Because of that traceable linkage, operational architectures are able to capture 
operational requirements in a way such that design specifications can be consistently 
traced to those requirements. 


The relationship between architectures and analyses using a variety of techniques 15 
only beginning to be explored. The close and mutually beneficial relationship between 
architecture and analysis need to be emphasized and strengthened. 


Architectures: Keys to Realizing Potential 


Must be able to access, reuse and manipulate data across a wide range of 
stakeholders 


° Users 

5 Developers 

° Acquirers 
Must integrate human factors into architecture: People are an integral 
part of architectures 
Need to represent threat more overtly to include analysis of adversary as 
well as friendly (interactive) 
Tools are critical 

° Better visualization 

“ Greater flexibility 

* More usable by the operational subject matter experts 
Must be able to better articulate the “so what” (operational impacts) of 
architectural analysis to warfighters and to decision makers 


Keys to Realizing Potential: 


In order to realize the full potential of architectures, the various stakeholders from 
system users to developers and acquirers, must be able to access the architecture data, 
manipulate it and reuse it in a variety of ways. 


Current architecture constructs do not overtly recognize and portray the impact of 
human factors. Because people are an integral part of any architecture, architecture 
constructs need to be expanded to more specifically represent human factors. 


Threat also needs to be represented more overtly in architectures. 


Getting data from a hardcopy text presentation to a structured softcopy presentation is 
essential and will enable the use of automated tool. The use of tools is critical to 
provide better visualization, greater flexibility and to make the data more usable to 
operational subject manner experts. 


Architecture description developers must perceive that their primary objective is not 
just the accumulation and presentation of the architecture data but the analysis of that 
data to identify the "so what", i.e. the operational impacts and to better articulate those 
impacts to warfighters and decision makers. 


Architectures: Measures of Merit 


Still very immature, promising efforts at early stages 
Must be able to establish quantifiable relationship between 
measures of performance and operational measures of 
effectiveness/outcome 


Measures of Merit: 


Currently there are no well-conceived and realized measures of merit associated with 
architectures. 


There are three aspects of architectures which potentially could be measured: 
(1) how well the architecture description has been developed, 

(2) to what extent is the architecture description being used and 

(3) the value of the architecture in achieving C4ISR goals and objectives. 


The third aspect is considered the most meaningful and the most difficult to quantify. 
It is the aspect most addressed in ongoing efforts. A metric for this aspect must be 
able to establish a quantifiable relationship between measures of performance for 
individual systems and operational measures of mission effectiveness associated with 
the architecture's system-of-systems. 


The community appears to just be in the initial stage of addressing metrics for 
architectures. While current efforts, such as those by John Hopkins, Boeing and 
OASD (C3)), are very immature but they show promise. 


The ability to measure the value of a specified architecture in achieving mission 
effectiveness is key to architecture-centric analysis. 
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Architectures: Assessment 


ΝΞ es Seats _ Human Factors 
Problem Structuring ne ae | Collaborative | ee ᾿Ξ : 
᾿ Development ο 


(Defined Methodolog: ἜΣ 


Robustness δὲ Flexibility 


_ Definition. Accessibility 

"Availability: a eee 
4 ‘Acquisition ~ Survivability (IA) 

: ——~ Analysis 


“Quality of Ὁ 


| Description- 


"Potential: 
Relevance © 


Assessment: 


The above graphic provides a color-coded assessment of the current state of the art for 
architectures. There is a well-defined methodology in place -- the C4JSR Architecture 
Framework has been mandated for use in DoD. 


While there is some capability to portray human factors via architectures, it is not well defined. 
In terms of the human factor aspect of architecture development, there is a need for a greater 
ability for collaborative development. 


Scenarios can be portrayed in architectures but additional robustness and flexibility is needed. 
Measures of Merit are only just now beginning to be addressed. 

While there are a variety of tools available, a coherent tool set is yet to be achieved. 
Architecture data is well defined, but there is no centrally-managed or maintained architecture 
data repository. Thus gathering data for architecture descriptions generally requires a great 


deal of primary collection effort. 


Architectures have the capability of reducing risk in acquisition efforts and supporting analysis 
of information assurance, but generally are not used to the extent possible. 


Developers are generally quite good at presenting the architecture data in a report but are 
frequently not successful in communicating the "so what" of an architecture. 
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Architectures: Summary 


Architectures are an evolving discipline 
- part science and part art 
Methodology well defined but still evolving 
e Potential for architecture-centric analysis 
- structured data repository 
- automated tools 
Need greater emphasis on the “so what” 
e Measures of Merit still in very early stages but some 
promising beginnings 


Summary: 


Architectures are an evolving discipline which has reached a certain level of maturity 
but will continue to evolve. Developing architecture descriptions requires both science 
and art. 


While the methodology is well defined in the C4ISR Architecture Methodology, it will 
continue to evolve based on use across DoD. 


There is great potential for architecture-centric analysis but a structured data repository 
and automated tools are critical to achieve that potential. 


Architectures must not be static descriptions but require greater emphasis of the "so 
what." 


Measures of Merit are in very early stages but there are promising beginnings across 
the community. 
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MILITARY OPERATIGNS RESEARCH SOCIETY 


Analyzing C4ISR for 2010 


Analytical Techniques and Tools 


This is the Final Out brief from the Analytical Techniques and Tools Working Group. Dr 
Al Brandstein was the Chair and Dr Roy Rice was the Co-chair. Members of the 
Working group are listed in the Attendees list. Special thanks goes to Mr Steve Ritacco 
and Captain Geof Maron for their superb work in recording the discussions and preparing 
this briefing. 


Analytical Techniques and Tools: 
Outline 


9 Introduction 
¢ Taxonomy 
e Summary 


Outline: 


Preparatory remarks are included in the Introduction. This details our mission and 
objectives for our Working Group. It also outlines the context by which we focused our 
discussions. Next is a detail of the taxonomy we devised to frame how we discussed the 
uses, misuses, deficiencies and opportunities for improvement for our analysis 


methodologies and tools. Finally, we summarize our two days of discussions with a list 
of findings. 


Analytical Techniques and Tools: 
Mission / Objectives 


Understand elements of C4ISR 


9 Discuss wide range of analytical methodologies 
᾽ν Focus on Decision Issues 
. Match methodologies to issues 


Determine deficiencies/opportunities 


Mission / Objectives: 


Our first objective was to come to some common understanding of the elements of 
C4ISR. This was from an operational, analytical, and modeling standpoint. This allowed 
us to focus on the analysis needs. Then we received several briefings covering a wide 
range of C4ISR topics and analysis approaches. With this range of applicable 
methodologies in mind, we discussed the various decision issues or questions that 
decision makers are asking now, or will probably be interested in, in the future. 


Our next effort was to match some of the relevant methodologies to the particular 
decision issues. This then allowed us to discuss some of the specific deficiencies or 
opportunities to improve the state-of-the-art. 


Analytical Techniques and Tools: 
Actions of Group 


Methodologies 
— JWARS; NETWARS; Optimization; Complex Adaptive 
Systems; Systems Dynamics; etc. 
— Discussions 
Develop taxonomy of Decision Issues 
Rate the “Starr”? Chart 
Glean pearls of wisdom 


Actions of Group: 


The first day of the workshop, we heard presentations from several attendees on 
analytical methodologies they are using, modeling efforts they are involved in, and 
current/future capabilities. These were briefings by: 


- LTC Dan Maxwell - WARS 

- LTC Pat Dye - NETWARS 

- Roy Rice - C4ISR Optimization using the Sensor Platform Allocation Model 
(SPAM) 

- Capt Geoff Maron - an application of Complex Adaptive Systems (CAS) 

- Charlie Hall - Linking C4ISR to Military Effectiveness 


- Steve Upton - Generative Analysis: Using Composable Agent-based Simulations for 
Future C4ISR Concept Exploration 


- Steve Kolek - Battlefield Communications Network Modeling 
- Robie Samanta Roy - Analyzing C4ISR on Strike Mission Performance 
- Evan Ellis - Systems Dynamics Modeling to Represent C4ISR 


The discussions that followed these presentations led us to develop a taxonomy by which 
to address many of the Decision Issues. This provided a context by which we rated the 
various areas of the chart Dr Starr provided as a guideline in his plenary presentation. 


Analytical Techniques and Tools: 
Context of C4ISR Modeling 


95 C4ISR “system” as 
— Independent variable 
— Dependent variable 
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Context of C4ISR Modeling: 


The discussions quickly led us to conclude that our analyses of relevant C4ISR “systems” 
includes C4ISR “systems” as independent variables and dependent variables. What we 
mean here is that we often perform analyses of the specific elements of C4ISR. We 
compare alternative sensors, platforms, architectures, and their performance. This is 
where C4ISR “systems” could be considered as independent variables. There are also the 
cases where we analyze other parts of our application of military force (forces, tactics, 
doctrine, etc.) where C4ISR “systems” must play but are not the analytical topic of 
interest. They must be represented and accounted for properly in our analyses and 
models. This is where they would be considered dependent variables. 


In that context, we used the depiction here to represent the elements of the C4ISR 
“system” from an operational, analytical, and modeling perspective. 


Analytical Techniques and Tools: 
Areas of C4ISR Analysis 


Decision Makers 


Category of 
Decision 


Issues 


Next Slide 
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Analytical Techniques and Tools: 
Within Each Area (Cube) 


Specific Questions 


Analysis 
Issues 


Tools/techniques 
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Analytical Techniques and Tools: 
Acquisition Slice 


Objectives 


Force Structure 
Cognitive Behavior 
CONOPS 


Architecture 
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Vulnerabilities 
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Analytical Techniques and Tools: 
Operational Slice 


Objectives 
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Analytical Techniques and Tools: 
POM / Budget Slice 


Objectives 


Force Structure 
Cognitive Behavior 
CONOPS 
Architecture 
Vulnerabilities 


Trade Space 
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Analytical Techniques and Tools: 
Acquisition / Force Structure 


Force Structure 


e MTW ἃ SSC 
~ Room for improvement 
- Scenarios rated Red, Data rated Red 
. OOTW 
- Data lacking 


- Gaping void below high level documents 
- Tools cannot be found even if objectives are ID’d 
- No architecture exists forinteragency C?(NGO ἃ GO) 
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Analytical Techniques and Tools: 
Acquisition / Cognitive Behavior 


Cognitive Behavior 


e Hard to get cognitive behavior effects 
Too recent 
Exploratory 


We don’t understand the process 
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Analytical Techniques and Tools: 
Acquisition / CONOPS 


CONOPS 


MTW /SSC 
— Doctrine is wired into the tools 


- C4 deficiencies exist within the tools 
e OOTW 
- Tools are lacking 


Analytical Techniques and Tools: 
Acquisition / Architecture 


Architecture 


e Data is difficult to obtain 
— EspeciallyinOOTW = (NGO and coalition) 
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Analytical Techniques and Tools: 
Acquisition / Vulnerabilities 


Vulnerabilities 


e Computer security / information warfare tools are required 


— Not much interest in this area 
— We have little experience or basis for creating these tools 


Analytical Techniques and Tools: 
Acquisition / Trade Space 


~ Data lacking 
— Gaping void below high level documents 
— No architecture exists for interagency C 2(NGO & GO) 
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Analytical Techniques and Tools: 
Operational / Force Structure 


OOTW 
Force Structure 


Greenish tint to the yellow 

— This is the area in which we have the most expertise 
Don’t model ambiguity well 

— We know we don’t do this 
Data remains a problem 

— Availability 

~ Abstraction 
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Analytical Techniques and Tools: 
Operational / Cognitive Behavior 


Cognitive Behavior 


e This area has significant potential 
— Effects of stress 

e Continue development of tools in this area 
— Measures to exploit our experiences 

e Need data collection methodology 
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Analytical Techniques and Tools: 
Operational / CONOPS 


CONOPS 


e We know methodologies 


e Don’t have the resolution in our tools to provide sufficient insights 
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Analytical Techniques and Tools: 
Operational / Architecture 


Architecture 


e Requires concentration of near term investment 
e Measures of Merit required to feed to operational decision makers 
e Tools need to be dynamic 


~ Economic pressures 
- COTS impact on force structure 
¢ Commercial satellite systems, ... 
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Analytical Techniques and Tools: 
Operational / Vulnerabilities 


Vulnerabilities 


e Need to be able to study the sensitivities of our architectures to our 
assumptions 
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Analytical Techniques and Tools: 
POM/Budget / Force Structure 


Force Structure 


Low level MoM acceptable, but aggregate measures are poor 
Scenarios fair for high levels, poor for low levels 
Tools are good at mission level 


Tools are poor at the joint level, across missions, and among services 
Data is fair 
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Analytical Techniques and Tools: 
POM/Budget / Cognitive Behavior 


Cognitive Behavior 


Ability in area is in infancy stages 
e Data is poor 


e Tools are poor 


Measures are fair 
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Analytical Techniques and Tools: 
POM/Budget / CONOPS 


CONOPS 


e CONOPS specified enough for budgeting purposes 
— Consideration of broader scenarios is poor 


e Tools may be poor, but data and measures are good 
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Analytical Techniques and Tools: 
POM/Budeet / Architecture 


Architecture 


No agreement on MoMs across scenarios, services, and different C4ISR 
systems 


e Tools are good 


Data 1s good to fair 
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Analytical Techniques and Tools: 
POM/Budget / Vulnerabilities 


Vulnerabilities 


Questions about levels of redundancy and levels of fitness are not well 
described 


Tools that show the effects of non-linear interactions and the ability to 
measure their impact on complex systems are lacking. 


Data is poor 
MOMs are poor 


Tools are fair 
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Analytical Techniques and Tools: 
POM/Budget / Trade Space 


Trade Space 


Good to fair within mission and functional area 
Poor to fair between missions and services 
MoMs fair 

Tools fair 


Data poor 
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Analytical Techniques and Tools: 
Matching Issues and Methodologies 


Force Structure 
Cam paign Level Sim 


Architecture ~ Mission Level Sim 
Engagement Level Sim 
CONOPS Optimization 
Game theory 


Technology -> New Doctrine/ 


. ic ᾿ 
tactics Dynamic Program ming 


Functional Decomposition 


\ 
Vulnerabilities Ps LES Systems Dynamics 


W argaming 
Trade Space Ζ oe. etc. 


Cognitive Behavior 
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Analytical Techniques and Tools: 
Coalitions 


Greater number of potential partners in PE than in specific MTWs 
Interoperable C4I required for coalitions 


Reliance on coalition wartime capabilities requires peacetime investment 
in (41 


Coalition interoperability improves war fighting capability 


Coalitions: 


These are conclusions about coalitions that resulted from discussion of PE in the broader 
context. 
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Analytical Techniques and Tools: 
Key Issues and Questions 


- After Cold War, the spectrum of potential conflict has broadened 


- The probability of occurrence of events on this spectrum may be 
differently distributed 


- Data collection and analysis of that issue should follow 


- Naval Postgraduate School research suggests major benefits ($10Bs) in 
savings on oil prices by prompt deployment of naval forces in crises. Can the 
scale of benefits for deployment of C4ISR be similarly identified (e.g., 
AWACS deployment impacting prices of drugs from Colombia)? 


Key Issues and Questions: 


1. The nature of the world, the threat, has changed drastically since the end of the cold 
war. The distribution of risk of warfare is quite different, with more uncertainty 
about the probabilities of specific threats. There is a need to collect data and develop 
the distribution of events on the current and future spectra of conflict. 


2. The Naval Postgraduate School study identified significant potential savings on oil 


prices by early deployment of naval forces. It is possible that early deployment of 
C4ISR systems could provide similar benefits. 
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Analytical Techniques and Tools: 
Key Issues and Questions (cont’d) 


3. The less an event looks like war, the less able we are to articulate its 
governing processes. C4ISR in PE requires drawing in political 
constraints/considerations to a degree greater than for other points on the 
spectrum. This suggests peculiar data requirements for military/C4ISR 
commanders in PE settings. 


. For PE events, the cyclical process (define MoM, measure sensitivity, collect 
data, evaluate data, report) needs to also capture critical C4ISR data and 
effects. 
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Recommendations: 


Analytical Techniques and Tools: 
Recommendations 


Develop broadly-accepted TOR for PE: definitions, objectives, goals, 
distinguishing relationships from 
OOTW to MTW 


Develop CSOP for PE 
Develop MoMs and data collection checklists for PE 


Expand JULLS formats and databases to include C4ISR data 


Recognizing PE as an important mission area, with attendant principles and 
distinguishing relationships, will assist in identifying the peculiar tasks of PE and 
garnering funding for their successful undertaking. 


Common Standard Operating Procedures must account for the many differences 
(historical, cultural, financial, educational, etc.) that exist in the many countries that 
we engage during PE operations. “One size fits all” is not the answer. We need to 
KNOW our PE “targets” much better and tailor each engagement to that particular 
country. Specific C4ISR exchanges are good candidate elements for tailored PE. 


Expansion of the Joint Uniform Lessons Learned database to include C4ISR data and 
information will help long term analysis of C4ISR issues. 
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Analytical Techniques and Tools: 
Recommendations (cont’d) 


9 Study the impact of deploying C4ISR in peacetime 
— Assess utility of C4ISR deployment on its own 


— Assess how early C4ISR deployment facilitates early combat power by 
follow-on forces 


Recommendations (cont’d): ! 


The results of such a study, successfully completed, should help determine the value | 
added by C4ISR systems. The study should also address how a robust C4ISR exchange | 
program helps build coalitions and therefore serves as a significant deterrent. | 
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Analytical Techniques and Tools: 
Assessment of Current State-of-the-Practice 


Human Factors & © 
_ Organization” 


Problem Structuring 


Assessment of Current State-of-the-Practice: 


Scenarios represent the only good news on this slide. In all other areas, assessment of the 
current state-of-the-practice is discouraging. There is much work to be done to develop 


measures of effectiveness and tools and to collect data relevant to proper analysis of 
C4ISR issues related to PE. 


G-34 


Analytical Techniques and Tools: 


Summary 
KEY ISSUES 


® Since the cold war, the spectrum of potential conflict has broadened. The 
probability of occurrence of events on this spectrum may be differently 
distributed. Data collection and analysis of that issue should follow. 


¢ The less an event looks like war, the less able we are to articulate its governing 
processes. C4ISR in PE requires drawing in political constraints/ 
considerations to a degree greater than for other points on the spectrum. This 
suggests peculiar data requirements for military/ C4ISR commanders in PE 
settings. 


¢ For PE events, the cyclical process (define MoM, measure sensitivity, collect 
data, evaluate data, report) needs to also capture critical C4ISR data and 
effects. 


RECOMMENDATIONS 


© Develop broadly-accepted TOR for PE: definitions, objectives, goals, 
distinguishing relationships from OOTW to MTW 


© Develop CSOP for PE 
© Develop MoMs and data collection checklists for PE 
¢ Expand JULLS formats and databases to include C4ISR data 


© Study the impact of deploying C4ISR in peacetime. Assess the utility of 
C4ISR deployment on its own. Assess how early C4ISR deployment facilitates 
early combat power by follow-on forces. 


Summary: 


KEY ISSUES 
¢ The nature of the world, and the threat, has changed since the end of the Cold 

| War. The distribution of risk of warfare is quite different, with more 
uncertainty about the probabilities of specific threats. There is a need to 
collect data and develop the distribution of events on the current and future 
spectra of conflict. 

RECOMMENDATIONS 

- Recognizing PE as an important mission area, with attendant principles and 
distinguishing relationships, will assist in identifying the peculiar tasks of PE, 
and gamering funding for their successful undertaking. 

¢ Common Standard Operating Procedures must account for the many 
differences (historical, cultural, financial, educational, etc.) that exist in the 
many countries that we engage during PE operations. “One size fits all” is not 
the answer. We need to KNOW our PE “targets” much better, and tailor each 
engagement to that particular country. Specific C4ISR exchanges are good 
candidate elements for tailored PE. 

- Expansion of the Joint Uniform Lessons Learned (JULLS) database to include 
C4ISR data and information will help long term analysis of C4ISR issues. 
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Analyzing C4ISR for 2010 
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Synthesis Group 
Synthesis Team 


Major Theater Warfare (MTW) 

— Denny Baer, Logicon 

— Jerry Kotchka, Boeing 
Smaller Scale Contingencies 
(SSC) 

— Dave Alberts, OASD(C3]1) 
Operations Other Than War 
(OOTW) 

— Russ Richards, MITRE 
Peacetime Engagement 

—- Clay Thomas, USAF 

— Bill Kemple, NPS 


Infrastructure Assurance 
(IA) 
— Stuart Starr, 
MITRE 
Information Architectures 
— Larry Wiener, DoN 
Analytical Techniques and 
Tools 


— Denis Clements, 
GRCI 


— Jay Kistler, USN 
— Steve Myer, USA 


Floaters 
— Bob Eberth, USNR 
— Pat Peterson, USN 


Synthesis Team: 


The Synthesis Team was created to develop a holistic view of the deliberations of all of 
the Working Groups participating in the Workshop. Consistent with that goal, a team 
was selected that included representation from OSD, each of the Services, industry, 
FFRDCs and academia. Selected members of the Synthesis Team actively participated in 
each of the Working Groups (see the Viewgraph for the specific assignments). The 
Synthesis Team convened periodically during the course of the Workshop to discuss and 
compare the evolving findings and recommendations of each Working Group. Drawing 
on those deliberations, the Synthesis Team developed some broad observations about the 
nature of the problem and formulated a set of cross-cutting findings and 
recommendations. These products are presented and discussed below. 
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Synthesis Group 
Broad Observations 


There is broad variability in the “maturity” of the mission areas; i.e., most 
mature to least mature: 

-- MTW 

-- SSC 

- OOTW 

-- Peacetime Engagement 

-- Infrastructure Assurance 
Selected mission areas (e.g.. OQOTW) subsume sub-missions that are 
qualitatively different (e.g., HA/DR, PK/PE) 


Broad Observations: 


As a consequence of monitoring the deliberations of the individual Mission Oriented 
Working Groups it became clear that there was broad variability in the maturity of the 
individual mission areas (e.g., level of domain knowledge; available tools and data; 
experience in analyzing the C4ISR systems and processes associated with those 
missions). Based on the perceptions of the Synthesis Team, it was concluded that one 
could rank order the missions on a qualitative “maturity scale” as follows: Major Theater 
War, Smaller Scale Contingency, Operations Other Than War, Peacetime Engagement 
and Infrastructure Assurance. 


The Synthesis Team observed further that many of these mission areas subsume 
qualitatively different sub-missions. For example, Operations Other Than War subsumes 
Humanitarian Assistance, Disaster Relief, Peace Keeping and Peace Enforcement. These 
sub-missions are so different in their nature and maturity that it can be misleading to 
make overarching statements about the overall mission area. 


Synthesis Group 
Major Issues 


Mission Area Major Issues 


Few accepted data sources; Modeling Red C4ISR; Existing data 
does not map to DPG scenarios 


MoMs need sufficient pre-definition 


Data; Softness of MoMs (public opinion); OOTWs are not all the 
same 
Definition 

njrastructure Data (much of it is Blue data); Scope of problem/ability to 


Assurance decompose meaningfully; Higher level MoMs Dealing with the 
interactive nature of the problem 


Info Arch How does one score an architecture and communicate the results 
appropriately? 


Tools & Tech Cognitive modeling; Broad array of missions; Size of the trade 
space; Representation of Red data 


Major Issues: 


As the Synthesis Team monitored the deliberations of the individual Working Groups it 
identified several major issues that were raised by those Working Groups. The 
associated viewgraph briefly summarizes those major issues. Note that although there 
are considerable differences in those issues, several cross-cutting issues are apparent. 
These cross-cutting issues involve difficulties associated with data and measures of merit 


(MoMs). These specific cross-cutting issues will be highlighted in the findings and 
recommendations that follow. 


H-4 


Synthesis Group 
Aggregate Assessment 


ΝΗ R: 


Aggregate Assessment: 


In his plenary presentation, Stuart Starr introduced a template that was developed in the 
forthcoming NATO Code of Best Practice (COBP) for Assessing Command and 
Control. [Note: for information purposes, a brief summary of the highlights of the 
NATO COBP is included as Annex A]. That template identified the life cycle of the C2 
assessment process, emphasizing the key steps that are followed by leading practitioners 
of C2 assessment. Each of the Working Groups used this template to assess the current 
state of the health of C2 assessment for their mission area. The associated viewgraph 
provides an aggregate assessment, qualitatively integrated across the results of the 
individual Working Group assessments. It can be seen that two of the critical steps (1.e., 
data, tools) were assessed as being poor (“Red”), five of the steps (i.e., problem 
structuring, treatment of human factors/organization, measures of merit, treatment of risk 
and uncertainty, report documentation and availability) were assessed as being 
significantly deficient (“orange”), and one step (i.e., scenario generation) was assessed 
as being fair (“yellow”). It is notable that, on aggregate, none of the steps were assessed 
as being good (‘green’’). 


These observations suggest that there is widespread concern about the community’s 
ability to adhere to the processes in a strawman COBP. These results led the Synthesis 
Group to highlight findings and recommendations in five areas: COBP, three key 
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elements of the candidate COBP and the overall relationship between the producer and 
consumer of C2 analyses. These findings and recommendations are summarized below. 


Synthesis Group 
Code of Best Practices (COBP): 
Findings 


e Findings 


— A community-endorsed COBP is needed to provide a sound 
foundation for 


© structuring, 


© addressing the C2 assessment problem, in all its dimensions 


Code of Best Practices (COBP) — Findings: 


The Synthesis Working Group concluded that the NATO COBP represented an 
interesting initial step towards developing a sound foundation for C2 assessment. 
However, that product was restricted in its scope to assessing C2 in the context of 
conventional warfare. As this Workshop revealed, the problem of analyzing C4ISR for 
2010 will involve a host of additional mission areas, which pose unique problems for the 
C2 assessment community. Thus, an expanded, community-endorsed COBP is needed to 
support the C2 assessment problem, in all its dimensions. 
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Synthesis Group 
Code of Best Practices (COBP): 
Recommendations 


Recommendations: 
e An assessment should be made of the NATO COBP to see if it meets 
the needs of the US C2 assessment community (Action: MORS) 
e If the assessment endorses the COBP, it should be 


— broadly disseminated to the C2 assessment community 
(Action: MORS) 


employed in defense universities to educate young analysts 

(Action: NPS, AFIT) 

e Efforts should be supported to extend the NATO COBP to address 
New World Disorder issues (Action: SAS Panel-015) 


Code of Best Practices (COBP) ~ Recommendations: 


Although the preliminary review of the NATO COBP was promising, it is recommended 
that a more thorough, in-depth assessment of the product should be undertaken to see if 
it meets (at least some of) the needs of the US C2 analysis community. MORS is in the 
best position of any US organization to organize and execute such an assessment. 


If such an assessment is undertaken and it endorses the NATO COBP as a preliminary 
product, the following steps should be pursued. First, the product should be disseminated 
broadly to the C2 assessment community, along with any caveats that emerged through 
the assessment process. MORS has an efficient distribution network for making such 
information broadly available. Second, consideration should be given to employing the 
product in defense universities, such as the Naval Postgraduate School (NPS) and the 
Air Force Institute of Technology (AFIT), to educate young analysts. 


In view of the limited scope of the NATO COBP, efforts should be supported to extend 
the product to address New World Disorder issues. NATO is forming Studies, Analysis 
and Simulation (SAS) Panel-015, to explore the feasibility of extending the initial COBP 
to deal with issues associated with OOTW and Information Operations. Those efforts 
Should be supported by MORS (e.g., by providing: representatives to the deliberations; 
appropriate briefings/case studies to the participants; a peer-review of the panel’s 
product) to ensure that the resulting efforts are of greatest utility to the US C2 
assessment community. 
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Synthesis Group 
Measures of Merit (MoMs): Findings 


e Findings 
— MORS’ concept of a hierarchy of MoMs appears valid 


~~ However, 
e the hierarchy must be extended to deal with the needs of New 
World Disorder missions 
. MoMs are lacking to support the effective assessment, 
comparison of architectural options 


Measures of Merit -- Findings: 


Over the last thirteen years, MORS has conducted several workshops on the issue of 
developing Measures of Merit (MoMs) for command and control assessments [Ref. 1, 
2]. Those workshops proposed the concept of a hierarchy of MoMs that would range 
from measures of system performance through measures of mission effectiveness. [Note: 
see Figure 3 in the appended paper].The various mission-oriented working groups at this 
workshop appeared to support this conceptual approach to the problem. However, it was 
concluded that the hierarchy must be extended to deal with the needs of New World 
Disorder missions. 


The Architecture Working Group also observed that it lacked MoMs to support the 
effective assessment and comparison of architectural options. In particular, it called for 
MoMs that would relate system architectural characteristics to operational outcomes. 


Synthesis Group 
Measures of Merit (MoMs): Recommendations 


e Recommendations 


—— Formulate upper levels of the MoM hierarchy to reflect effectiveness 
of C4ISR with respect to 
e political-institutional stability (e.g., OOTW, SSC) 
e public confidence in institutions (e.g., IA) 


— Explore options to develop MoMs that can be used to assess and 
compare architectural options (e.g., relate system architectural 
characteristics to operational outcomes) 

(Action: OSD host key decision makers; MORS facilitate) 


Measures of Merit -- Recommendations: 


To respond to the needs of New World Disorder missions, efforts are needed to 
formulate the upper levels of the MoM hierarchy. For example, in the mission areas of 
Smaller Scale Contingencies and Operations Other Than War, measures are needed to 
characterize political-institutional-social stability (e.g., are the bodies of government 
functioning effectively; are educational institutions operative; are children playing on 
soccer fields). Similarly, in the area of Infrastructure Assurance, there is a need for 
MoMs that reflect public confidence in key infrastructures (e.g., confidence in the 
finance and banking sector as measured by the value of the Dow Jones Industrial 
Average and the value of the dollar versus the Euro). These needs suggest the need for 


MORS to convene a workshop on expanded C4ISR MoMs for New World Disorder 
missions. 


To satisfy the needs of the Architectural community, steps should be taken to explore 
options to develop MoMs that can be used to assess and compare architectural options. 
One possible course of action would call for OSD to convene an Integrated Process 
Team (IPT) of stakeholder organizations to address this issue. MORS could provide the 
facilitators to conduct such an event. 


Synthesis Group 
Data: Findings 


Findings 
— Data issues were highlighted by each Working Group 


—- Specific concerns varied by Group 
¢ Dealing with data appropriate to future operations (6.5. MTW) 

Acquiring needed Blue system data (e.g., TA) 
Characterizing Red C4ISR, infrastructure (e.g., MTW, IA) 
Gaining access to data in a timely fashion (All) 
Collecting and making better use of data emerging from 
experimentation (e.g., MT'W, SSC, OOTW) 
‘‘,.-there is no V&C worthy of the name” (Eberth) 


Data — Findings: 


A decade ago, at a MORS Workshop on Simulation Technology 1997 (SIMTECH 97), 
Dr. Walt LaBerge observed that “without data we are nothing” [Ref. 3]. That thought 
clearly prevailed in this workshop and data issues were highlighted by each Working 
Group. The specific concerns varied by Working Group and included the following 
challenges: acquiring the data appropriate to some currently ill-defined future military 
operation; acquiring the Blue system data needed to perform vulnerability assessments 
of critical infrastructures; characterizing Red’s C4ISR systems and _ critical 
infrastructures; gaining access to needed data in a timely fashion; collecting and making 
better use of data emerging from Service and Joint Advanced War fighting Experiments. 
One of the members of the Synthesis Working Group also observed that there is “... no 
validation and certification worthy of the name.” 
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Synthesis Group 
Data: Recommendations 


“Establish an institutional mechanism for acquiring, VV&Cing, 
transforming, storing and making accessible key data 


τ Draw on the capabilities and resources of key organizations (e.g., 
DSC, PA&E (e.g., JDS), DMSO (e.g., MSOSA), J6 (e.g., JDIICS, 
NETWARS)) (Action: OASD(C31)) 


Data -- Recommendations: 


The depth and breadth of these data concerns are such that a new institutional 
mechanism is needed to deal with the totality of the problem. This mechanism should 
deal with the full life cycle of the data problem, to include data acquisition, verification, 
validation and certification (VV&C), transformation (into a form that is useful to the C2 
assessment community), storage and access. Currently, there are a number of 
organizations that are involved with significant facets of the problem. These include 
OSD’s C4ISR Decision Support Center (DSC), PA&E’s Joint Data System (JDS), 
Defense Modeling and Simulation Office (DMSO) Modeling and Simulation 
Operational Support Activities (MSOSA) and J6’s Joint Defense Information 
Infrastructure Control System (JDIICS) and NETWARS activities. OASD(C3I) should 
take the lead in forging these fragmented efforts into a complete, mutually reinforcing 
solution to the total problem. 
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Synthesis Group 
Tools and Their Application: Findings 


© Findings 


- There is no single tool (or class of tools) that can satisfy C2 
assessment needs adequately 


Tools and Their Application — Findings: 


Many of the speakers at the workshop focused their remarks on modeling and simulation 
and its potential role in C2 assessment. It is clear, however, that no single class of tool 
(and certainly no single tool within a class) can satisfy C2 assessment needs adequately. 


Synthesis Group 
Tools and Their Application: Recommendations 


Recommendations 
— Encourage the use of a broad mix of tools that are well suited to the 
critical issues of interest; e.g., 
® analyses, based on basic principles 
° structured expert elicitation tools 


* creative applications of “New Sciences” to deal with Complex 
Adaptive Systems 


¢ multi-functional tools (e.g., support to analysis and rehearsal) 


Assiduously pursue efforts to perform responsible levels of VV&A 
(Action: MORS) 


Apply these tools iteratively (e.g., generalization of the Model-Test- 
Model paradigm) 


Tools and Their Application — Recommendations: 


There is a broad spectrum of tools that a C2 analyst can chose from, that vary with 
respect to the time to create and apply them, the cost to create and apply them and their 
credibility. [Note: see Figure 4 in the appended paper]. It is important that C2 assessors 
be aware that this broad mix of tools exists (beyond the “usual suspects” of M&S) and 
be conversant about their strengths and weaknesses. In particular, where appropriate, C2 
assessments should take better advantage of analyses, based on basic operations research 
and physical principles; structured expert elicitation tools (e.g., group ware); applications 
of the “New Sciences” to deal with Complex Adaptive Systems; and multifunctional 
tools (e.g., tools that can support analyses and mission rehearsal). In all instances, 
assessors should assiduously pursue efforts to perform responsible levels of verification, 
validation and accreditation (VV&A) for the tools selected. MORS should lead the way 
in determining what is a “responsible level of VV&A.” [Note: it is anticipated that 
SIMVAL 99 will shed light on this issue]. 


Because of the strengths and weaknesses of these tools, it is important to select and 
orchestrate a complementary set of tools. In the past, this concept has been limited to the 
Model-Test-Model paradigm. However, it would be appropriate to generalize that 
concept to incorporate a broader set of tool types. For example, it might be desirable to 
consider a “expert elicitation-real world experience-test-model” paradigm, that expands 
the set of tools that are orchestrated. 
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Synthesis Group 
Provider-Consumer Relations: Findings 


e Findings 


— Providers are failing to make their analyses transparent, 
understandable 


— Consumers are frequently not “educated customers” 


Provider-Consumer Relations — Findings: 


Many of the Working Groups observed that relations between the providers and 
consumers of C2 assessment are strained. They noted that many providers, by relying on 
complex, opaque M&sS are failing to make their analyses transparent and understandable 
(e.g., “Why did I come to that conclusion? My model told me!’’). Similarly, consumers 
are frequently not “educated customers.” For example, they fail to articulate issues in a 
way that is amenable to responsible C2 assessment, or they fail to give the provider 
adequate resources to either create the needed tools or to perform the assessment. 


Synthesis Group 
Provider-Consumer Relations: Recommendations 


e Recommendations 


"— There should be a “contract”? between providers and consumers, 
recognizing the mutual responsibilities of both sides 


~~ A COBP should be developed (perhaps in the form of a check list) to 
help educate consumers about the attributes of sound C4ISR 
assessment 
(Action: MORS) 


Provider-Consumer Relations — Recommendations: 


This issue was recognized and addressed at the MORS Mini-Symposium on “Quick 
Reaction Analysis Requirements and Methodologies (QRAM)” [Ref. 4]. The Synthesis 
Working Group at that workshop recognized that there should be a “contract” between 
providers and consumers, recognizing the mutual responsibilities of both sides. It is 
recommended that the draft “contract” be updated and refined to reflect the lessons 
learned from this workshop. 


As a further step, a COBP should be developed (perhaps in the form of a checklist) to 
help educate consumers about the attributes of sound C4ISR assessment. This “check 
list” could be patterned after the COBP that the AIAA Information & C2 System 
Technical Committee is developing for OSD on C2 experimentation. 
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Synthesis Group 
Summary 


Good News 
— Weare beginning to understand 
Φ the importance of C4ISR 


© the practices that should be followed to perform credible C4ISR 
analyses 


® =... and what we don’t understand! 
e Bad News 


— Currently, we generally do not do a satisfying job in performing C4ISR 
analyses 
Worse News 


— With the arrival of the New World Disorder, the C4ISR assessment 
problem is getting substantially more difficult 
e Better News 


— If the recommendations of the workshop are implemented, it will provide 
the foundation for the C4ISR community to work collaboratively to attack 
the most critical of our shortfalls 


Summary: 


The Synthesis Group discerned news on C2 assessment that ran the gamut from 
good to bad to worse to better. 


The good news is that we are beginning to understand a great deal more about the 
C2 assessment problem. We are starting to recognize the importance of C4ISR in 
the context of the many missions that DoD must perform; the practices that 
should be followed to perform credible C4ISR analyses; ... and what we don’t yet 
understand! 


The bad news is that currently, we generally do not do a satisfying job in 
performing C4ISR analyses. Although there are counter-examples to this 
statement, it is recognized that most studies of conventional conflict fail to live up 
to the COBP generated by NATO’s SAS Panel-002. 


The worse news is that the C2 assessment problem is getting substantially more 
difficult. We are confronted with a host of New World Disorder missions and 
issues for which we lack the key elements of effective C2 analyses (particularly 
relevant data and tools). In addition, these issues are in a constant state of flux, 
making it difficult to get traction on the problem. One of the most dramatic 
examples of this challenge is the Infrastructure Assurance problem, where the 
infrastructures and the threats to them are changing dramatically. 
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Although these are daunting challenges, there is some better news that is implicit 
in this workshop itself. If the recommendations of the workshop are implemented 
successfully, it will have provided the foundation for the C4ISR community to 
work collaboratively to attack the most critical of our shortfalls. 
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MNGAS TERMS OF REFERENCE 
Analyzing C4ISR for 2010 Workshop 


Background 


The Revolution in Military Affairs has already begun to transform our command and control 
procedures and architectures, as we struggle to adapt our institutions to the realities of the 
Information Age. Recent seminal defense planning documents, e.g., Joint Vision 2010, 
Quadrennial Defense Review, and the National Defense Panel report, have highlighted the critical 
role of information technology-based C4ISR for future US forces. A broader spectrum of threats 
and operations, a smaller more capable force, and an information technology-based society 
require greatly improved C4ISR capabilities. In our resource-constrained environment, however, 
decision makers face tough choices in determining the allocation of investment between C4ISR 
and the other contributors to force effectiveness (e.g., platforms and weapons, modernization, 
sustainment, logistics). The analytical community must be able to provide cogent analyses to 
support these decisions. What has been missing heretofore is the ability to assess effectively the 
relative contribution of C4ISR to force effectiveness vis-a-vis other factors. Past analyses — with 
their component metrics, methodologies, and tools — have not been adequate for this new 
emphasis. The analytic community must now posture itself to address the analysis of C4ISR for 
2010 and beyond. 


This special meeting, comprising analysts and C4ISR subject matter experts, will concentrate on 
the improvement of analysis of C4ISR for a US force and doctrine heavily reliant on information 
technologies. To achieve this end, participants will enumerate potential decision maker issues and 
define the appropriate analytic metrics, methodologies, and tools for C4ISR analysis. These 
issues will be addressed by working groups covering the spectrum of military operations, as well 
as information infrastructures and leading-edge analytical techniques. 


Goals and Objectives 


Goals. This workshop will afford the military OR community an opportunity to achieve the 
following goals: (1) share information on the current state of C4ISR analysis; (2) determine the 
appropriate metrics and methodologies for analysis of C4ISR for 2010; and (3) define key areas 
of improvement for the application of analysis in support of decision making on C4ISR 
investments and warfighting utility. The results will provide insights into how to advance the 
ability of the OR community to conduct analysis of C4ISR in support of Joint Vision 2010 and 
beyond. 


Objectives. The objectives are to: 


Enumerate the relevant issues pertaining to the analysis of C4ISR in 2010 

Identify metrics that are sensitive to the effects of C4ISR on force-level effectiveness 
Assess methodologies to analyze and quantify the effectiveness of C4ISR 

Evaluate appropriate tools to measure the benefit of C4ISR 

Discuss the requirements for and employment of advanced tools, methods, and research. 


Approach 


In order to achieve these goals and objectives, the subject of C4ISR analysis will be examined in 
a number of different contexts. A working group will examine C4ISR analysis within each 
contextual framework. Working groups will be asked to address some baseline issues (as a 
Starting point) and provide specific results. The working group contexts, baseline issues, and 
desired results are detailed below. 


Working Groups. The workshop attendees will be organized into working groups (18- 20 
members each) to examine the analysis of C4ISR within these specific focus areas: 


Major Theater War 

Smaller Scale Contingencies 
Operations Other Than War 
Infrastructure Assurance 
Peacetime Engagement 
Information Architectures 
Analytical Techniques and Tools 
Synthesis 


A. Major Theater War. Co-chairs: Dr. Mark Youngren and Chuck Taylor. This working 
group will focus on C4ISR within the context of large-scale, joint and combined operations 
against a technologically sophisticated adversary. These highly complex operations will require 
fusion of all-source intelligence, seamless integration of sensors, platforms, and command 
organizations, and intense logistic support to allow a greater number of operational tasks to be 
accomplished faster. An important consideration for this group is ascertaining our ability to 
determine the relative contributions of C4ISR and platforms/weapons to force effectiveness. 


B. Smaller Scale Contingencies. Chair: Dick Hayes. This working group will focus on C4ISR 
in the context of smaller scale operations, including "show of force", crisis response, raids, 
opposed (or potentially opposed) noncombatant evacuations, sanction enforcement, special 
operations, and counterterrorism. These operations may require covert Intelligence-Surveillance- 
Reconnaissance (ISR) and communications, flexible contingency planning, command and control 
responsive to national as well as tactical concerns, and highly portable, reconfigurable, and secure 
joint support systems. They may also involve interagency or coalition issues. 


C. Operations Other Than War. Co-chairs: Dr. Cy Staniec, John Furman and Terry Prosser. 
This working group will focus on C4ISR within the context of OOTW. In order to bolster 
regional stability, US forces are increasingly likely to be involved in disaster relief, humanitarian 
assistance, peacekeeping and peace enforcement, and similar non-traditional military missions. In 
these settings, coordination with non-DoD entities (e.g., State Department), non-governmental 


organizations (e.g., International Red Cross), and host nation governments will entail unique 
C4ISR requirements. 


D. Infrastructure Assurance. Co-chairs: Thomas Bozek and Frank Ruggeri. This working 
group will focus on defense of key elements of national infrastructure. DoD is dependent on a 
complex framework of networks and systems to provide the continual flow of essential goods and 
services, including information, electrical power, finance and banking, transportation, fuel 
distribution, health care, and emergency services. The military infrastructure is characterized by 
increasingly interdependent mix of defense and commercial services; reliance on commercial 
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services; interdependency among services driven by changing business practices and applied 
information technology; and vulnerabilities to failures and disruptions from asymmetric attacks, 
malicious intrusion, inadvertent error, and natural disasters. 


E. Peacetime Engagement. Chair: Bruce Powers. This working group will focus on the 
peacetime presence of US forces throughout the globe as a means to safeguard our vital national 
interests, promote regional stability, deter potential adversaries, and build cooperative relations 
with the world’s most influential countries. In maintaining a substantial overseas presence, we 
will maintain the capability to conduct the full range of military operations through a combination 
of permanently stationed forces, rotationally deployed forces, temporarily deployed forces, and 
infrastructure. 


F. Information Architectures. Chair: Patsy McGrady. This working group will focus on the 
implications of information architecture (system, technical, and operational) for the analysis of 
C4ISR. Included in this discussion will be information management (process) and infrastructure. 
Critical questions include how can analysis contribute to developing information management 
architectures and doctrines that will facilitate getting the right information to the right place at the 
right time and can analysis be flexible enough to evaluate alternative processes. 


G. Analytical Techniques and Tools. Co-chairs: Dr. Alfred Brandstein and Dr. Roy Rice. 
This working group will focus on the leading edge of analysis, specifically the latest 
developments in modeling (e.g., genetic algorithms) and their applicability to the analysis of 
C4ISR. Define the appropriate use of various methodological constructs (process, case-based, 
network-oriented, etc.) to conduct C4ISR analysis. Address the adequacy of supporting 
algorithms, simulations, experiments, and exercises. 


H. Synthesis Working Group. Chair: Dr. Stuart Starr. This working group will take a broader 
view, determine any commonality of issues and concerns across the spectrum of working groups, 
and provide an integrating perspective. Two specific issues of interest for this group will be the 
utility of the Joint Warfare System (ΟΥ̓ ΑΚ 5) and the important of information dominance. 


Tasking. Working groups will be directed to address the following questions and concerns in the 
following priority order: 


Characterize C4ISR within the working group focus area. Structure and decompose the 
elements of C4ISR. Describe the scenario space, key factors, and concerns that 
potentially affect the analysis and representation of C4ISR. Comment on the adequacy of 
current mission area CONOPS to support assessment of the C4ISR contribution in that 
area. 


Define the relative worth of C4ISR. What are the relative contributions of C4ISR 
(information), firepower (weapons and platforms), and force employment strategies (e.g., 
Dominant Maneuver)? How can analysts make meaningful tradeoff analyses? What are 
the breakpoints in C4ISR capability and spread? 


Develop and recommend Measures of Merit. Characterize and organize — for primary 
(warfare) and secondary (C4ISR) functions — the applicable measures, metrics, and 
variables including Measures of Performance (MOPs), Measures of Effectiveness 
(MOEs), Measures of Force Effectiveness (MOFEs), etc. Assess how well these metrics 
can establish the value of C4ISR relative to other contributors to force effectiveness. 
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Evaluate how current-or new-metrics can be used to calculate Return on Investment 
(ROT) for C4ISR. 


Identify and describe tools. Identify and describe models, simulations, decision support 
systems, and other C4ISR-applicable software tools in terms of capability, status, 
effectiveness, and ability to contribute to the development of operational and technical 
architectures. Identify needs for new software tools to support advanced C4ISR analysis. 


Analyze, synthesize, and infer. Identify common issues and concerns across each of the 
four topic areas and draw generalized conclusions and make relevant recommendations. 


Sequence of Events, The morning of the first day will be devoted to a plenary session for 
speakers. The speakers will provide differing perspectives on the problem of C4ISR analysis. The 
keynote speaker will be asked to provide the perspective of the decision maker as the ultimate 
customer of analytical products. The second speaker(s) will provide an operational perspective, 
specifically the view of various CINCs. The last speaker will provide a technical perspective. At 
the end of these sessions, the Chair will provide a charge to the working groups to ensure all 
participants share a common vision of the products required. The afternoon will be devoted to 
working group sessions under the supervision of the working group chairs to begin discussions on 
the designated evaluation issue. At the end of this day, there will be a social event (mixer). The 

_ second day will be devoted to parallel working group sessions to further develop their 
discussions. There will be a luncheon speaker on this day. The morning of the third day will 
consist of final working sessions to finalize their presentations and written products, followed by 
a plenary sessions for working group presentations. 


Attendees 


Attendance will be controlled via invitation. Attendees will include invited experts from OSD, all 
Services, the Joint Staff, Federally Funded Research and Development Centers, operational 
commanders, and DoD contractors. Workshop chairs will control membership of their sessions in 
conjunction with the Organizing Committee. Attendance will be limited to 150 people. 


Products 
There will be four specific products generated as a result of this workshop: 


e A scripted briefing for the MORS Sponsors addressing the workshop objectives, findings 
conclusions, and recommendations. 

e A proceedings containing summaries of all sessions and copies of appropriate briefing 
slides and presentations. 
A general session presentation for the 67th MORSS. 
A PHALANX article 
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Proponents 
Proponents for the workshop are: 


Army Deputy Under Secretary of the Army (Operations Research) (DUSA(OR)) 
Navy Director of Program Resource Appraisal, Chief of Naval Operations (N81) 
Director for Command and Control, DCS, Air and Space Operations, HQUSAF 


Planning and Organizing Committee 


General Chair: 


Technical Chair: 


Deputy Chair: 


Synthesis Chair: 


Site Coordinator: 


Administrative Coordinators: 


Organizing Committee: 


RADM Robert Nutwell, USN 

Deputy Director 

Space, Information Warfare Command and Control 
Directorate (N6B) 

Chief of Naval Operations 

2000 Navy Pentagon 

Washington, DC 20350-2000 

(703) 601-1200 


Dr. Russ Richards 

The MITRE Co 

1820 Dolley Madison Bivd 
McLean, VA 22102 

(703) 883-6787 


CAPT Jay Kistler, USN 

Director, Navy Modeling and Simulation 
Management Office (N6M) 

Chief of Naval Operations 

2000 Navy Pentagon 

Washington, DC 20350-2000 

(703) 601-1482 


Dr. Stuart Starr 

The MITRE Co 

1820 Dolley Madison Blvd 
McLean, VA 22102-3481 
(703) 883-5494 


LTC Jerry Nye, USA 
U.S. Army War College 
Carlisle Barracks 
Carlisle, PA 17013-5099 
(717) 245-4353 


Richard Wiles and Natalie Addison 
MORS, Suite 202 

101 S. Whiting Street 

Alexandria, VA 22304 

(703) 751-7290 


Army - TBA 

Air Force - Clayton J. Thomas, FS, AFSAA 
Marines — Dr. Alfred Brandstein 

OSD - Dr. Jackie Henningsen, FS 

Joint Staff - Peter Byrne 

NRO - TBA 

DISA - TBA 
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At large: Dr. Dave Alberts (ASD(C3D) 
Dr. Tom Cedel (TASC) 
Chartier Chris (OSD) 

Barry Dysart (N6M - SAIC) 
Dean Free (N81) 

Paul Hobbes (N81 — SAIC) 
Susan Iwanski (SPA) 

Dr. Jerry Kotchka, Boeing 
Dr. Ivar Oswalt (Kapos) 
Vince Roske (J-8) 

John Kelsey (TASC) 

Doug Sizelove (ODUSA(OR)) 
Dr. Larry Wiener (N6C5) 
LTC Patrick Vye, USA (J-6) 


Administration 


Name: Analysis of C4ISR 2010 

Dates: 27-29 October 1998 

Location: Center for Strategic Leadership, Carlisle Barracks, PA 
Fee: Federal Government Employees $175; All others $350 
Attendance: Limited to 150 

Classification: SECRET 


Agenda 


Tuesday - 27 October 


0700 
0800 
0810 
0820 


0840 


Registration and Continental Breakfast 
MORS President's Welcome Denny Baer 


Welcome by Commandant MG Robert Scales, USA 


Army War College 


Overview of Workshop RADM Nutwell, USN 


Introduction of Keynote Speaker 


Bliss Hall 


Keynote: Mr. Art Money, ASD(C3I) 


"The Decision Maker's Perspective" 
"The Operators Perspective" TBA 
Break 


“C3]I — The Past, Present and Future” Dr. Stuart Starr, MITRE 


Transportation from Bliss Hall to Collins Hall 
Catered Lunch 

Working Group Sessions 

Break 

Mixer 


Wednesday - 28 October 


1315 
1330 
1430 
1700 


Continental Breakfast 

Working Group Discussions 

Transportation from Collins Hall to Letort View Community Center 
Buffet Lunch With Guest Speaker: 


"Federated analysis of C4ISR in Warfare" Mr.Charles Taylor, 
OASD(C3D/DSC 

Transportation from Letort View Community Center to Collins Hall 

Working Group Sessions 

Break 

Adjourn 


Thursday - 29 October 


0715 
0800 


1000 
1200 
1300 
1315 
1330 
1345 
1400 
1415 
1430 
1445 
1500 
1515 
1530 
1600 


Continental Breakfast 

Working Group: Prepare Briefings to 

Summarize Deliberations 

Plenary Session: "Blackhawk Down" Mark Bowden 
Catered Lunch 

Plenary Session: Working Group Reports WG Chairs 
Major Theater War 

Smaller Scale Contingencies 

Operations Other Than War 

Peacetime Engagement 

Infrastructure Assurance 

Break 

Information Architectures 

Analytical Techniques and Tools 

Synthesis 


Closing Remarks RADM Nutwell 


Adjourn Workshop 


Collins Hall 


Letort View 
Comm. Center 


Collins Hall 


Letort View 
Comm. Center 


Collins Hall 


Collins Hall 


Letort View 
Comm. Center 
Bliss Hall 


Milestone Plan 


Date "D" 
30 Jan -270 
31 Mar -210 
30 Apr -180 
30 May -150 
9Jun = -140 
29 Jun -120 
29 Jul =-90 
3 Aug -85 
8 Aug -80 
13 Aug -75 
28 Aug -60 
7Sep -50 
12Sep -45 
17 Sep -40 
2QOct -25 
27-29 Oct 
26 Nov 30 
26 Dec 60 
25 Jan 90 
24Feb 120 
26 Mar 150 
25 Apr 180 


Task 
Initiate draft TOR procedure 
Select tentative dates 


Provide "For Comment" draft of TOR to Sponsors, 


VP(MO) and other interested organizations and 
individuals for review 

Revise TOR 

Circulate initial draft TOR to MORS Office and 
committee for comments 


Solicit candidate for working group chairs 
Circulate final draft TOR to MORS Office and 


proponents for concurrence and to other Sponsors 


and organizations for information 
Approve TOR, program chair, budget and fees 


Select working group chairs 

Mail ACP, if appropriate 

Select organization to be invited and 
prepare letter inviting nominations 

Mail invitations to nominating organizations 
Select read-ahead material 


Select nominations or requests for applications 
Select invitees 
Assign nominees to working groups 
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